Sep 16 21:22:18.667069 ip-10-0-136-118 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 21:22:19.085252 ip-10-0-136-118 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 21:22:19.085252 ip-10-0-136-118 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 21:22:19.085252 ip-10-0-136-118 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 21:22:19.085252 ip-10-0-136-118 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 21:22:19.085252 ip-10-0-136-118 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 21:22:19.085931 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.085323    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 21:22:19.090378 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090351    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:22:19.090378 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090375    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:22:19.090378 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090379    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:22:19.090378 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090382    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090385    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090391    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090395    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090398    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090402    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090405    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090408    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090410    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090413    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090416    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090419    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090421    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090425    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090430    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090433    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090436    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090439    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090441    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090444    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:22:19.090570 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090447    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090450    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090452    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090455    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090458    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090461    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090464    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090469    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090472    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090475    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090478    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090481    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090483    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090486    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090490    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090493    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090496    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090499    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090501    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090504    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:22:19.091054 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090509    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090511    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090514    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090517    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090519    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090523    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090526    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090529    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090531    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090534    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090536    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090539    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090544    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090547    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090550    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090552    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090557    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090561    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090564    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:22:19.091553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090568    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090571    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090574    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090577    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090580    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090585    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090589    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090592    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090594    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090597    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090600    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090603    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090606    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090610    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090613    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090617    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090620    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090622    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090628    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:22:19.092031 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090631    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090633    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090636    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090639    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090644    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090647    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090651    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090655    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090658    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090660    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090666    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090669    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090672    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090675    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090678    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090681    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090683    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.090686    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092207    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092215    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:22:19.092489 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092218    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092221    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092224    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092226    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092230    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092232    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092237    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092241    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092247    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092250    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092253    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092255    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092258    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092261    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092264    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092267    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092270    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092272    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092275    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:22:19.093077 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092278    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092283    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092286    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092288    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092291    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092294    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092297    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092300    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092303    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092305    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092308    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092311    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092314    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092317    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092322    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092325    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092328    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092331    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092334    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:22:19.093577 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092337    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092339    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092342    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092345    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092349    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092353    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092356    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092361    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092365    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092368    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092371    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092374    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092377    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092380    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092382    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092387    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092391    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092394    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092398    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:22:19.094082 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092404    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092407    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092410    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092413    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092416    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092419    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092422    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092424    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092427    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092430    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092432    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092435    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092441    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092444    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092446    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092449    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092452    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092454    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092457    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092460    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:22:19.094553 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092463    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092466    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092469    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092472    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092474    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092480    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092483    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092486    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092490    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092493    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092496    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092499    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092502    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092505    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092507    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092510    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092514    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092519    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092522    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.092525    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:22:19.095038 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093160    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093165    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093168    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093171    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093174    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093177    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093181    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093184    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093186    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093189    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093192    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093197    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093200    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093203    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093206    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093209    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093212    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093214    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093217    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093220    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:22:19.095517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093224    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093228    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093232    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093238    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093241    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093244    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093247    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093250    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093252    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093255    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093258    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093260    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093263    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093265    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093268    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093273    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093277    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093279    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093282    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093286    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:22:19.096020 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093289    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093292    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093295    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093299    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093302    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093307    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093310    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093316    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093318    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093321    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093324    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093327    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093329    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093333    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093335    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093338    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093342    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093345    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:22:19.096512 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093350    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093352    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093355    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093358    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093360    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093363    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093365    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093368    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093371    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093374    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093376    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093379    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093381    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093387    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093390    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093392    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093395    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093397    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093400    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093404    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:22:19.096969 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093407    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093410    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093413    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093415    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093418    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093421    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093426    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093429    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093431    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093434    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093437    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093439    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093442    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093445    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093447    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093450    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093453    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093455    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093458    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093463    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:22:19.097446 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.093465    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.093899    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.093959    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.093976    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.093986    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.093993    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094002    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094011    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094022    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094027    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094034    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094041    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094048    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094055    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094062    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094069    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094076    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094082    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094089    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094098    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094105    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094112    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094118    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094125    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 21:22:19.097958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094134    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094140    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094147    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094153    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094157    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094160    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094163    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094166    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094169    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094173    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094177    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094180    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094184    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094187    2563 flags.go:64] FLAG: --enable-server="true"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094190    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094194    2563 flags.go:64] FLAG: --event-burst="100"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094197    2563 flags.go:64] FLAG: --event-qps="50"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094201    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094204    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094207    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094211    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094214    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094217    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094220    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094223    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 21:22:19.098526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094226    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094231    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094234    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094237    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094240    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094242    2563 flags.go:64] FLAG: --feature-gates=""
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094251    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094254    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094257    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094260    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094263    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094266    2563 flags.go:64] FLAG: --help="false"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094269    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094272    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094275    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094278    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094281    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094286    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094289    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094292    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094295    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094298    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094301    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094304    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 21:22:19.099179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094307    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094310    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094313    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094316    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094319    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094322    2563 flags.go:64] FLAG: --lock-file=""
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094325    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094327    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094330    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094336    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094340    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094343    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094346    2563 flags.go:64] FLAG: --logging-format="text"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094349    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094352    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094355    2563 flags.go:64] FLAG: --manifest-url=""
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094358    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094362    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094365    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094370    2563 flags.go:64] FLAG: --max-pods="110"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094373    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094376    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094379    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094382    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094385    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 21:22:19.099786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094388    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094391    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094400    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094403    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094406    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094410    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094413    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094418    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094421    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094424    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094427    2563 flags.go:64] FLAG: --port="10250"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094430    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094433    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e9f4cfd6c49c3eb7"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094436    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094439    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094442    2563 flags.go:64] FLAG: --register-node="true"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094445    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094449    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094452    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094455    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094458    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094462    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094465    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094468    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094471    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 21:22:19.100406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094474    2563 flags.go:64] FLAG: --runonce="false"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094476    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094479    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094483    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094486    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094489    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094492    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094496    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094499    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094501    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094504    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094507    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094510    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094513    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094516    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094519    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094524    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094527    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094530    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094534    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094537    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094540    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094543    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094546    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094548    2563 flags.go:64] FLAG: --v="2"
Sep 16 21:22:19.101023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094553    2563 flags.go:64] FLAG: --version="false"
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094558    2563 flags.go:64] FLAG: --vmodule=""
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094562    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.094565    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094683    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094688    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094691    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094694    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094697    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094700    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094703    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094707    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094709    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094712    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094717    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094720    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094723    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094725    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094728    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094730    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:22:19.101631 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094733    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094735    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094738    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094741    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094744    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094746    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094749    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094752    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094770    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094775    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094780    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094783    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094787    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094789    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094793    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094796    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094799    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094802    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094804    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:22:19.102151 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094807    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094809    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094812    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094815    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094817    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094821    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094823    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094827    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094830    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094833    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094836    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094839    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094841    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094844    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094846    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094849    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094851    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094854    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094856    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094859    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:22:19.102643 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094861    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094864    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094866    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094869    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094871    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094874    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094877    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094879    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094893    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094895    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094898    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094901    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094903    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094906    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094908    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094911    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094913    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094916    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094918    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094922    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:22:19.103154 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094925    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094929    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094932    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094935    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094938    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094941    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094944    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094948    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094950    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094953    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094956    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094960    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094963    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094966    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094969    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094972    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094975    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094978    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094980    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:22:19.103658 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094983    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:22:19.104171 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094985    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:22:19.104171 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094989    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:22:19.104171 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094992    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:22:19.104171 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.094994    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:22:19.104171 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.095771    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 21:22:19.104171 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.103558    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 21:22:19.104171 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.103574    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 21:22:19.104526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.104515    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 21:22:19.104557 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.104526    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 21:22:19.104608 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104600    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:22:19.104608 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104608    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104611    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104615    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104619    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104624    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104627    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104631    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104635    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104638    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104642    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104645    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104648    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104651    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104654    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104658    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104661    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104663    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104667    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:22:19.104661 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104670    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104673    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104676    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104679    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104681    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104687    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104690    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104693    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104695    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104698    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104701    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104703    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104706    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104709    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104712    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104714    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104717    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104720    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104722    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104725    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:22:19.105112 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104727    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104730    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104732    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104735    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104737    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104740    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104742    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104745    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104747    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104750    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104753    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104766    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104769    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104771    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104775    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104778    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104781    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104784    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104787    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104789    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:22:19.105604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104792    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104795    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104798    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104800    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104803    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104805    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104808    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104811    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104813    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104816    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104820    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104823    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104826    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104829    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104832    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104834    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104837    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104840    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104842    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:22:19.106115 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104845    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104847    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104850    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104852    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104855    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104857    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104860    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104863    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104866    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104869    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104871    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104874    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104881    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104884    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104886    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104889    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104892    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104894    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104897    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104899    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:22:19.106604 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.104902    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:22:19.107102 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.104907    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 21:22:19.107102 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105020    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:22:19.107102 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105024    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:22:19.107102 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105027    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:22:19.107102 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105030    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:22:19.107102 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105032    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:22:19.107102 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105035    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:22:19.107102 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105038    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:22:19.107102 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105040    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:22:19.107102 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105043    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:22:19.107102 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105046    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:22:19.107102 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105048    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:22:19.107102 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105050    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:22:19.107102 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105053    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:22:19.107102 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105055    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105058    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105061    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105063    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105066    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105069    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105072    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105074    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105076    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105079    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105082    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105085    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105087    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105090    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105092    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105095    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105097    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105100    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105102    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105104    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:22:19.107477 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105107    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105109    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105112    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105114    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105117    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105120    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105122    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105125    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105128    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105130    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105133    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105135    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105138    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105140    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105143    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105147    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105150    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105153    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105156    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:22:19.107979 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105159    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105162    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105165    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105168    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105172    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105175    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105178    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105180    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105183    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105185    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105188    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105191    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105194    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105196    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105199    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105201    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105204    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105206    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105209    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105211    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:22:19.108443 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105214    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105216    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105219    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105221    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105224    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105226    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105229    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105231    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105234    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105236    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105239    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105243    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105246    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105249    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105252    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105255    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105258    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105261    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105263    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:22:19.108940 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105266    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:22:19.109401 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105268    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:22:19.109401 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105271    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:22:19.109401 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105274    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:22:19.109401 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105276    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:22:19.109401 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105278    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:22:19.109401 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105281    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:22:19.109401 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:19.105284    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:22:19.109401 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.105289    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 21:22:19.109401 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.105407    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 21:22:19.109401 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.107820    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 21:22:19.109401 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.108921    2563 server.go:1019] "Starting client certificate rotation"
Sep 16 21:22:19.109401 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.109019    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 21:22:19.109815 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.109803    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 21:22:19.112698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.112684    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 21:22:19.114254 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.114233    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 21:22:19.123944 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.123926    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 16 21:22:19.128970 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.128943    2563 log.go:25] "Validated CRI v1 image API"
Sep 16 21:22:19.129826 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.129808    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 21:22:19.132448 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.132422    2563 fs.go:135] Filesystem UUIDs: map[4ca635fb-583e-4820-a696-30619c7f2165:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d5374adc-7877-400a-92b6-a9c7c182ff31:/dev/nvme0n1p3]
Sep 16 21:22:19.132500 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.132449    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 21:22:19.136911 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.136890    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 21:22:19.139659 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.139554    2563 manager.go:216] Machine: {Timestamp:2026-09-16 21:22:19.137959556 +0000 UTC m=+0.368067048 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099511 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20d6db745250cbd05923544e55cc60 SystemUUID:ec20d6db-7452-50cb-d059-23544e55cc60 BootID:edc84036-0df0-45c1-8aa4-efa52fd5789a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e2:36:24:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e2:36:24:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:c6:8a:c8:71:91 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 21:22:19.139659 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.139657    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 21:22:19.139770 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.139736    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 21:22:19.142669 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.142645    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 21:22:19.142817 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.142672    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-118.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 21:22:19.143557 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.143544    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 21:22:19.143594 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.143560    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 21:22:19.143594 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.143568    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 21:22:19.143594 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.143582    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 21:22:19.145021 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.145011    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 21:22:19.149661 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.149651    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 21:22:19.149796 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.149786    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 21:22:19.149832 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.149823    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 21:22:19.149863 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.149833    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 21:22:19.149863 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.149844    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 21:22:19.149863 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.149853    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 16 21:22:19.149863 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.149863    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 21:22:19.150250 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.150234    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 21:22:19.150335 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.150255    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 21:22:19.153289 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.153268    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 21:22:19.156190 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.156174    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 21:22:19.156247 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.156205    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 21:22:19.158480 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.158462    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 21:22:19.158559 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.158491    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 21:22:19.158559 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.158504    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 21:22:19.158559 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.158515    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 21:22:19.158559 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.158531    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 21:22:19.158559 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.158542    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 21:22:19.158559 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.158562    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 21:22:19.158735 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.158570    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 21:22:19.158735 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.158579    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 21:22:19.158735 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.158585    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 21:22:19.158735 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.158599    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 21:22:19.158735 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.158605    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 21:22:19.159438 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.159423    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 21:22:19.159438 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.159435    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 21:22:19.159860 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.159844    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mpvld"
Sep 16 21:22:19.160609 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.160583    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 21:22:19.160797 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.160772    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-118.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 21:22:19.163456 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.163439    2563 server.go:1268] "Started kubelet"
Sep 16 21:22:19.163567 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.163497    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 21:22:19.163680 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.163630    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 21:22:19.163752 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.163702    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 21:22:19.165104 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.165081    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 21:22:19.165189 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.165167    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mpvld"
Sep 16 21:22:19.165266 ip-10-0-136-118 systemd[1]: Started Kubernetes Kubelet.
Sep 16 21:22:19.166906 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.166874    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 21:22:19.170792 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.170441    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 21:22:19.171100 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.171083    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-118.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 21:22:19.174003 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.173979    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 21:22:19.174525 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.171156    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-118.ec2.internal.18d5ea062b17eff3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-118.ec2.internal,UID:ip-10-0-136-118.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-118.ec2.internal,},FirstTimestamp:2026-09-16 21:22:19.163332595 +0000 UTC m=+0.393440164,LastTimestamp:2026-09-16 21:22:19.163332595 +0000 UTC m=+0.393440164,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-118.ec2.internal,}"
Sep 16 21:22:19.174682 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.174664    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 21:22:19.175405 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.175302    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-118.ec2.internal\" not found"
Sep 16 21:22:19.175813 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.175791    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 21:22:19.175896 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.175793    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 21:22:19.175896 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.175837    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 21:22:19.176000 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.175906    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 21:22:19.176000 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.175916    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 21:22:19.176000 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.175961    2563 factory.go:55] Registering systemd factory
Sep 16 21:22:19.176000 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.175979    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 16 21:22:19.176209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.176196    2563 factory.go:153] Registering CRI-O factory
Sep 16 21:22:19.176242 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.176211    2563 factory.go:223] Registration of the crio container factory successfully
Sep 16 21:22:19.176280 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.176250    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 21:22:19.176280 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.176264    2563 factory.go:103] Registering Raw factory
Sep 16 21:22:19.176280 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.176278    2563 manager.go:1196] Started watching for new ooms in manager
Sep 16 21:22:19.177234 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.177221    2563 manager.go:349] Starting recovery of all containers
Sep 16 21:22:19.177772 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.177736    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:22:19.180563 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.180530    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-118.ec2.internal\" not found" node="ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.182751 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.182704    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 21:22:19.187703 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.187685    2563 manager.go:354] Recovery completed
Sep 16 21:22:19.193000 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.192989    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:22:19.196156 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.196134    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:22:19.196250 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.196164    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:22:19.196250 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.196175    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:22:19.196852 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.196838    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 21:22:19.196852 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.196852    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 21:22:19.196977 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.196865    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:22:19.196977 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.196869    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 21:22:19.197612 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.197596    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:22:19.197689 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.197625    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:22:19.197689 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.197641    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:22:19.199125 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.199110    2563 policy_none.go:49] "None policy: Start"
Sep 16 21:22:19.199125 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.199126    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 21:22:19.199214 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.199136    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 21:22:19.201100 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.201089    2563 policy_none.go:47] "Start"
Sep 16 21:22:19.226533 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.226357    2563 manager.go:337] "Starting Device Plugin manager"
Sep 16 21:22:19.226533 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.226523    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 21:22:19.252036 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.226537    2563 server.go:87] "Starting device plugin registration server"
Sep 16 21:22:19.252036 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.226751    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 21:22:19.252036 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.226776    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 21:22:19.252036 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.226873    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 21:22:19.252036 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.226982    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 21:22:19.252036 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.226994    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 21:22:19.252036 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.228772    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 21:22:19.252036 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.228823    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-118.ec2.internal\" not found"
Sep 16 21:22:19.271772 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.271731    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 21:22:19.271876 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.271799    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 21:22:19.271876 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.271825    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 21:22:19.271964 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.271910    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 21:22:19.275712 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.275688    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:22:19.326902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.326875    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:22:19.328360 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.328342    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:22:19.328449 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.328372    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:22:19.328449 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.328383    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:22:19.328449 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.328407    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.347072 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.347053    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.347137 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.347077    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-118.ec2.internal\": node \"ip-10-0-136-118.ec2.internal\" not found"
Sep 16 21:22:19.372291 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.372266    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-118.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal"]
Sep 16 21:22:19.372371 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.372326    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:22:19.376841 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.376824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/126583346e4bbf13f4d978c46ad24fd3-config\") pod \"kube-apiserver-proxy-ip-10-0-136-118.ec2.internal\" (UID: \"126583346e4bbf13f4d978c46ad24fd3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.378561 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.378545    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:22:19.378635 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.378576    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:22:19.378635 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.378586    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:22:19.380995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.380982    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:22:19.381089 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.381075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.381128 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.381103    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:22:19.381720 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.381704    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:22:19.381720 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.381712    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:22:19.381839 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.381733    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:22:19.381839 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.381743    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:22:19.381839 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.381733    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:22:19.381839 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.381796    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:22:19.383985 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.383972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.384039 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.383998    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:22:19.384568 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.384547    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-118.ec2.internal\" not found"
Sep 16 21:22:19.385576 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.385560    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:22:19.385667 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.385590    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:22:19.385667 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.385603    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:22:19.409460 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.409439    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-118.ec2.internal\" not found" node="ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.413780 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.413747    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-118.ec2.internal\" not found" node="ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.477591 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.477557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/126583346e4bbf13f4d978c46ad24fd3-config\") pod \"kube-apiserver-proxy-ip-10-0-136-118.ec2.internal\" (UID: \"126583346e4bbf13f4d978c46ad24fd3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.477591 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.477591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/196354e2e4c46cf552a666282e669d68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal\" (UID: \"196354e2e4c46cf552a666282e669d68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.477811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.477610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/196354e2e4c46cf552a666282e669d68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal\" (UID: \"196354e2e4c46cf552a666282e669d68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.477811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.477680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/126583346e4bbf13f4d978c46ad24fd3-config\") pod \"kube-apiserver-proxy-ip-10-0-136-118.ec2.internal\" (UID: \"126583346e4bbf13f4d978c46ad24fd3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.484625 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.484601    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-118.ec2.internal\" not found"
Sep 16 21:22:19.577996 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.577960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/196354e2e4c46cf552a666282e669d68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal\" (UID: \"196354e2e4c46cf552a666282e669d68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.577996 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.577989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/196354e2e4c46cf552a666282e669d68-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal\" (UID: \"196354e2e4c46cf552a666282e669d68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.578189 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.578037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/196354e2e4c46cf552a666282e669d68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal\" (UID: \"196354e2e4c46cf552a666282e669d68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.578189 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.578075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/196354e2e4c46cf552a666282e669d68-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal\" (UID: \"196354e2e4c46cf552a666282e669d68\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.585045 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.585019    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-118.ec2.internal\" not found"
Sep 16 21:22:19.686018 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.685927    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-118.ec2.internal\" not found"
Sep 16 21:22:19.712168 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.712137    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.717257 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.716978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal"
Sep 16 21:22:19.786387 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.786352    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-118.ec2.internal\" not found"
Sep 16 21:22:19.886952 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:19.886921    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-118.ec2.internal\" not found"
Sep 16 21:22:19.978079 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:19.978011    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:22:21.406133 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.406099    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-118.ec2.internal"
Sep 16 21:22:21.413717 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.413698    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 21:22:21.415296 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.415283    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal"
Sep 16 21:22:21.424532 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.424490    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 21:22:21.438663 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.438638    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 21:22:21.438810 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.438781    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 21:22:21.438926 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.438813    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 21:22:21.438926 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.438824    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 21:22:21.479885 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.479862    2563 apiserver.go:52] "Watching apiserver"
Sep 16 21:22:21.493956 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.493925    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 21:22:21.494294 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.494272    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5","openshift-dns/node-resolver-ftdzk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal","openshift-multus/multus-additional-cni-plugins-vrl4w","openshift-multus/network-metrics-daemon-4qxdd","openshift-network-operator/iptables-alerter-4smn4","openshift-ovn-kubernetes/ovnkube-node-lk5dz","kube-system/konnectivity-agent-s89hp","kube-system/kube-apiserver-proxy-ip-10-0-136-118.ec2.internal","openshift-cluster-node-tuning-operator/tuned-zxv2h","openshift-image-registry/node-ca-xfzln","openshift-multus/multus-7m8bq","openshift-network-diagnostics/network-check-target-9pdvd","kube-system/global-pull-secret-syncer-zxtz8"]
Sep 16 21:22:21.497220 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.497202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.499245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.499227    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 21:22:21.499755 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.499584    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 21:22:21.499755 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.499616    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 21:22:21.499755 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.499627    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 21:22:21.499983 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.499966    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:21.500139 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.500119    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 21:22:21.500329 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.500310    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 21:22:21.500454 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:21.500080    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:21.500796 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.500772    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lhjks\""
Sep 16 21:22:21.501840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.501794    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 21:17:20 +0000 UTC" deadline="2028-06-16 21:08:48.021366078 +0000 UTC"
Sep 16 21:22:21.501840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.501838    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15335h46m26.519531504s"
Sep 16 21:22:21.503148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.503129    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:21.503233 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:21.503193    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:21.504088 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.504073    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 21:22:21.505826 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.505809    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.507693 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.507669    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 21:22:21.507793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.507750    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 21:22:21.507873 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.507850    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9fflp\""
Sep 16 21:22:21.507981 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.507895    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 21:22:21.507981 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.507914    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 21:22:21.507981 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.507973    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 21:22:21.508463 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.508448    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:21.508542 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:21.508522    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:21.510336 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8800530f-c408-44d0-a430-bd6d8cd8d95c-dbus\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:21.510407 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/839fa7f2-7940-4702-8a23-d273fa25149e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.510407 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-slash\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510407 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-run-openvswitch\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510407 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/955b6bc5-4971-4cf9-bb64-01e237d331ae-env-overrides\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510550 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-node-log\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510550 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-run-netns\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510550 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/839fa7f2-7940-4702-8a23-d273fa25149e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.510550 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/839fa7f2-7940-4702-8a23-d273fa25149e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.510550 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510550 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/955b6bc5-4971-4cf9-bb64-01e237d331ae-ovnkube-config\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510550 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/955b6bc5-4971-4cf9-bb64-01e237d331ae-ovn-node-metrics-cert\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510550 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/955b6bc5-4971-4cf9-bb64-01e237d331ae-ovnkube-script-lib\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/839fa7f2-7940-4702-8a23-d273fa25149e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.510995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-etc-openvswitch\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-cni-bin\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/839fa7f2-7940-4702-8a23-d273fa25149e-system-cni-dir\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.510995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510644    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/839fa7f2-7940-4702-8a23-d273fa25149e-cnibin\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.510995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs\") pod \"network-metrics-daemon-4qxdd\" (UID: \"4254080d-20cf-49fc-a3ab-6ffbf08e4f44\") " pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:21.510995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcczn\" (UniqueName: \"kubernetes.io/projected/839fa7f2-7940-4702-8a23-d273fa25149e-kube-api-access-tcczn\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.510995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-systemd-units\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-run-systemd\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-var-lib-openvswitch\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-cni-netd\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510819    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-kubelet\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-log-socket\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.510995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:21.511610 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.510938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wngl2\" (UniqueName: \"kubernetes.io/projected/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-kube-api-access-wngl2\") pod \"network-metrics-daemon-4qxdd\" (UID: \"4254080d-20cf-49fc-a3ab-6ffbf08e4f44\") " pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:21.511610 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.511064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-run-ovn\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.511610 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.511098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wnfp\" (UniqueName: \"kubernetes.io/projected/955b6bc5-4971-4cf9-bb64-01e237d331ae-kube-api-access-7wnfp\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.511610 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.511123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvn7w\" (UniqueName: \"kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w\") pod \"network-check-target-9pdvd\" (UID: \"b6ce6503-957e-48b0-836a-699b9dd71a86\") " pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:21.511610 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.511147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8800530f-c408-44d0-a430-bd6d8cd8d95c-kubelet-config\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:21.511610 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.511169    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4smn4"
Sep 16 21:22:21.512763 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.512741    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 21:22:21.513098 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.513065    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 21:22:21.513098 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.513064    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 21:22:21.513238 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.513223    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 21:22:21.513554 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.513327    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9cc4r\""
Sep 16 21:22:21.513760 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.513742    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s89hp"
Sep 16 21:22:21.515717 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.515699    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lx29h\""
Sep 16 21:22:21.515800 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.515711    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 21:22:21.515800 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.515783    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 21:22:21.516556 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.516524    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.518397 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.518377    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 21:22:21.518490 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.518465    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 21:22:21.518540 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.518494    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 21:22:21.518540 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.518501    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sdprx\""
Sep 16 21:22:21.519157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.519140    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ftdzk"
Sep 16 21:22:21.520837 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.520821    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 21:22:21.520938 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.520893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n2nfl\""
Sep 16 21:22:21.520999 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.520937    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 21:22:21.521828 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.521812    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.523821 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.523803    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n5rx9\""
Sep 16 21:22:21.523821 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.523813    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 21:22:21.523965 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.523921    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 21:22:21.524593 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.524582    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xfzln"
Sep 16 21:22:21.526597 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.526581    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 21:22:21.526716 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.526625    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 21:22:21.526716 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.526691    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9ftm7\""
Sep 16 21:22:21.526716 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.526697    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 21:22:21.529922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.528017    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.531454 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.531435    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 21:22:21.531546 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.531498    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-22xjm\""
Sep 16 21:22:21.572904 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.572882    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k5tn6"
Sep 16 21:22:21.582128 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.582103    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k5tn6"
Sep 16 21:22:21.606327 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.606306    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 21:22:21.611428 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wnfp\" (UniqueName: \"kubernetes.io/projected/955b6bc5-4971-4cf9-bb64-01e237d331ae-kube-api-access-7wnfp\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.611527 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8800530f-c408-44d0-a430-bd6d8cd8d95c-kubelet-config\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:21.611527 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611452    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8800530f-c408-44d0-a430-bd6d8cd8d95c-dbus\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:21.611527 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-tuned\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.611527 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbmcf\" (UniqueName: \"kubernetes.io/projected/bd26ace6-2a22-4558-b175-ac19ec00836b-kube-api-access-rbmcf\") pod \"iptables-alerter-4smn4\" (UID: \"bd26ace6-2a22-4558-b175-ac19ec00836b\") " pod="openshift-network-operator/iptables-alerter-4smn4"
Sep 16 21:22:21.611527 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbrk8\" (UniqueName: \"kubernetes.io/projected/fd1d301e-815d-4471-a53a-b7a3bd452da8-kube-api-access-kbrk8\") pod \"node-ca-xfzln\" (UID: \"fd1d301e-815d-4471-a53a-b7a3bd452da8\") " pod="openshift-image-registry/node-ca-xfzln"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-run-k8s-cni-cncf-io\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-run-multus-certs\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8800530f-c408-44d0-a430-bd6d8cd8d95c-kubelet-config\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8800530f-c408-44d0-a430-bd6d8cd8d95c-dbus\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-slash\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-run-openvswitch\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/955b6bc5-4971-4cf9-bb64-01e237d331ae-env-overrides\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-node-log\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-run-netns\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/839fa7f2-7940-4702-8a23-d273fa25149e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611716    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-slash\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611733    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-kubernetes\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611765    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-node-log\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-sysctl-conf\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611765    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-run-netns\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-run-openvswitch\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.611811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611802    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-sys\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd26ace6-2a22-4558-b175-ac19ec00836b-host-slash\") pod \"iptables-alerter-4smn4\" (UID: \"bd26ace6-2a22-4558-b175-ac19ec00836b\") " pod="openshift-network-operator/iptables-alerter-4smn4"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611850    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-var-lib-cni-bin\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/955b6bc5-4971-4cf9-bb64-01e237d331ae-ovn-node-metrics-cert\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/955b6bc5-4971-4cf9-bb64-01e237d331ae-ovnkube-script-lib\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611961    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldmfc\" (UniqueName: \"kubernetes.io/projected/0f48f3a5-577d-426d-8288-75e7a451c50f-kube-api-access-ldmfc\") pod \"node-resolver-ftdzk\" (UID: \"0f48f3a5-577d-426d-8288-75e7a451c50f\") " pod="openshift-dns/node-resolver-ftdzk"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-systemd\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.611992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-lib-modules\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd26ace6-2a22-4558-b175-ac19ec00836b-iptables-alerter-script\") pod \"iptables-alerter-4smn4\" (UID: \"bd26ace6-2a22-4558-b175-ac19ec00836b\") " pod="openshift-network-operator/iptables-alerter-4smn4"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd1d301e-815d-4471-a53a-b7a3bd452da8-serviceca\") pod \"node-ca-xfzln\" (UID: \"fd1d301e-815d-4471-a53a-b7a3bd452da8\") " pod="openshift-image-registry/node-ca-xfzln"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-etc-openvswitch\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-etc-openvswitch\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-cni-bin\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/839fa7f2-7940-4702-8a23-d273fa25149e-cnibin\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-device-dir\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.612558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f48f3a5-577d-426d-8288-75e7a451c50f-hosts-file\") pod \"node-resolver-ftdzk\" (UID: \"0f48f3a5-577d-426d-8288-75e7a451c50f\") " pod="openshift-dns/node-resolver-ftdzk"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/955b6bc5-4971-4cf9-bb64-01e237d331ae-env-overrides\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-run\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/839fa7f2-7940-4702-8a23-d273fa25149e-cnibin\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-tmp\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-registration-dir\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-cni-bin\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612271    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nspsq\" (UniqueName: \"kubernetes.io/projected/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-kube-api-access-nspsq\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612303    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/839fa7f2-7940-4702-8a23-d273fa25149e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-systemd-units\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-cni-netd\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-systemd-units\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-cni-netd\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612412    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/371857b4-159c-47eb-897c-e71bb61469c7-konnectivity-ca\") pod \"konnectivity-agent-s89hp\" (UID: \"371857b4-159c-47eb-897c-e71bb61469c7\") " pod="kube-system/konnectivity-agent-s89hp"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612440    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/955b6bc5-4971-4cf9-bb64-01e237d331ae-ovnkube-script-lib\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.613433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612478    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-etc-selinux\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612534    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-var-lib-kubelet\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-multus-conf-dir\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-kubelet\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-log-socket\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wngl2\" (UniqueName: \"kubernetes.io/projected/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-kube-api-access-wngl2\") pod \"network-metrics-daemon-4qxdd\" (UID: \"4254080d-20cf-49fc-a3ab-6ffbf08e4f44\") " pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-host-kubelet\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612702    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-sysctl-d\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612715    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-log-socket\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:21.612752    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd1d301e-815d-4471-a53a-b7a3bd452da8-host\") pod \"node-ca-xfzln\" (UID: \"fd1d301e-815d-4471-a53a-b7a3bd452da8\") " pod="openshift-image-registry/node-ca-xfzln"
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:21.612817    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret podName:8800530f-c408-44d0-a430-bd6d8cd8d95c nodeName:}" failed. No retries permitted until 2026-09-16 21:22:22.112795715 +0000 UTC m=+2.013005709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret") pod "global-pull-secret-syncer-zxtz8" (UID: "8800530f-c408-44d0-a430-bd6d8cd8d95c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-system-cni-dir\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-run-ovn\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.614166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvn7w\" (UniqueName: \"kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w\") pod \"network-check-target-9pdvd\" (UID: \"b6ce6503-957e-48b0-836a-699b9dd71a86\") " pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/839fa7f2-7940-4702-8a23-d273fa25149e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.612922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-run-ovn\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-multus-cni-dir\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-hostroot\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/371857b4-159c-47eb-897c-e71bb61469c7-agent-certs\") pod \"konnectivity-agent-s89hp\" (UID: \"371857b4-159c-47eb-897c-e71bb61469c7\") " pod="kube-system/konnectivity-agent-s89hp"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-var-lib-kubelet\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-socket-dir\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613171    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhn2p\" (UniqueName: \"kubernetes.io/projected/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-kube-api-access-hhn2p\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/839fa7f2-7940-4702-8a23-d273fa25149e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-sys-fs\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-multus-socket-dir-parent\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613287    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-run-netns\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/955b6bc5-4971-4cf9-bb64-01e237d331ae-ovnkube-config\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/839fa7f2-7940-4702-8a23-d273fa25149e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f48f3a5-577d-426d-8288-75e7a451c50f-tmp-dir\") pod \"node-resolver-ftdzk\" (UID: \"0f48f3a5-577d-426d-8288-75e7a451c50f\") " pod="openshift-dns/node-resolver-ftdzk"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/839fa7f2-7940-4702-8a23-d273fa25149e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.614642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/839fa7f2-7940-4702-8a23-d273fa25149e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhpnz\" (UniqueName: \"kubernetes.io/projected/c56686e1-b7c9-4db3-9013-28e89905285e-kube-api-access-nhpnz\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-cnibin\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-cni-binary-copy\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-var-lib-cni-multus\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-etc-kubernetes\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/839fa7f2-7940-4702-8a23-d273fa25149e-system-cni-dir\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs\") pod \"network-metrics-daemon-4qxdd\" (UID: \"4254080d-20cf-49fc-a3ab-6ffbf08e4f44\") " pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcczn\" (UniqueName: \"kubernetes.io/projected/839fa7f2-7940-4702-8a23-d273fa25149e-kube-api-access-tcczn\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/839fa7f2-7940-4702-8a23-d273fa25149e-system-cni-dir\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-host\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:21.613697    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/955b6bc5-4971-4cf9-bb64-01e237d331ae-ovnkube-config\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/839fa7f2-7940-4702-8a23-d273fa25149e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-multus-daemon-config\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:21.613859    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs podName:4254080d-20cf-49fc-a3ab-6ffbf08e4f44 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:22.113845932 +0000 UTC m=+2.014055914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs") pod "network-metrics-daemon-4qxdd" (UID: "4254080d-20cf-49fc-a3ab-6ffbf08e4f44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:21.615157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-run-systemd\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.615682 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-var-lib-openvswitch\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.615682 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-run-systemd\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.615682 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-modprobe-d\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.615682 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.613991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955b6bc5-4971-4cf9-bb64-01e237d331ae-var-lib-openvswitch\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.615682 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.614003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-sysconfig\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.616345 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.616325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/955b6bc5-4971-4cf9-bb64-01e237d331ae-ovn-node-metrics-cert\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.621836 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.621807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wnfp\" (UniqueName: \"kubernetes.io/projected/955b6bc5-4971-4cf9-bb64-01e237d331ae-kube-api-access-7wnfp\") pod \"ovnkube-node-lk5dz\" (UID: \"955b6bc5-4971-4cf9-bb64-01e237d331ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.622213 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:21.622191    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:21.622213 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:21.622215    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:21.622346 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:21.622225    2563 projected.go:196] Error preparing data for projected volume kube-api-access-gvn7w for pod openshift-network-diagnostics/network-check-target-9pdvd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:21.622346 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:21.622269    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w podName:b6ce6503-957e-48b0-836a-699b9dd71a86 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:22.122257705 +0000 UTC m=+2.022467687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gvn7w" (UniqueName: "kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w") pod "network-check-target-9pdvd" (UID: "b6ce6503-957e-48b0-836a-699b9dd71a86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:21.623906 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.623885    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcczn\" (UniqueName: \"kubernetes.io/projected/839fa7f2-7940-4702-8a23-d273fa25149e-kube-api-access-tcczn\") pod \"multus-additional-cni-plugins-vrl4w\" (UID: \"839fa7f2-7940-4702-8a23-d273fa25149e\") " pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.624079 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.624062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wngl2\" (UniqueName: \"kubernetes.io/projected/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-kube-api-access-wngl2\") pod \"network-metrics-daemon-4qxdd\" (UID: \"4254080d-20cf-49fc-a3ab-6ffbf08e4f44\") " pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:21.680746 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:21.680693    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126583346e4bbf13f4d978c46ad24fd3.slice/crio-d4bfc5b682a1244e32c2c4939b1b074e00e084b78c7008beb160f3ca9798f101 WatchSource:0}: Error finding container d4bfc5b682a1244e32c2c4939b1b074e00e084b78c7008beb160f3ca9798f101: Status 404 returned error can't find the container with id d4bfc5b682a1244e32c2c4939b1b074e00e084b78c7008beb160f3ca9798f101
Sep 16 21:22:21.684379 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.684363    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 21:22:21.714792 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.714765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-tuned\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.714894 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.714796    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbmcf\" (UniqueName: \"kubernetes.io/projected/bd26ace6-2a22-4558-b175-ac19ec00836b-kube-api-access-rbmcf\") pod \"iptables-alerter-4smn4\" (UID: \"bd26ace6-2a22-4558-b175-ac19ec00836b\") " pod="openshift-network-operator/iptables-alerter-4smn4"
Sep 16 21:22:21.714894 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.714812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbrk8\" (UniqueName: \"kubernetes.io/projected/fd1d301e-815d-4471-a53a-b7a3bd452da8-kube-api-access-kbrk8\") pod \"node-ca-xfzln\" (UID: \"fd1d301e-815d-4471-a53a-b7a3bd452da8\") " pod="openshift-image-registry/node-ca-xfzln"
Sep 16 21:22:21.714894 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.714835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-run-k8s-cni-cncf-io\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.714894 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.714858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-run-multus-certs\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.714894 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.714883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-kubernetes\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.714910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-run-k8s-cni-cncf-io\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.714924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-run-multus-certs\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.714906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-sysctl-conf\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.714976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-sys\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd26ace6-2a22-4558-b175-ac19ec00836b-host-slash\") pod \"iptables-alerter-4smn4\" (UID: \"bd26ace6-2a22-4558-b175-ac19ec00836b\") " pod="openshift-network-operator/iptables-alerter-4smn4"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-var-lib-cni-bin\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-kubernetes\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-sysctl-conf\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldmfc\" (UniqueName: \"kubernetes.io/projected/0f48f3a5-577d-426d-8288-75e7a451c50f-kube-api-access-ldmfc\") pod \"node-resolver-ftdzk\" (UID: \"0f48f3a5-577d-426d-8288-75e7a451c50f\") " pod="openshift-dns/node-resolver-ftdzk"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-var-lib-cni-bin\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-systemd\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715095    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd26ace6-2a22-4558-b175-ac19ec00836b-host-slash\") pod \"iptables-alerter-4smn4\" (UID: \"bd26ace6-2a22-4558-b175-ac19ec00836b\") " pod="openshift-network-operator/iptables-alerter-4smn4"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-lib-modules\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-sys\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd26ace6-2a22-4558-b175-ac19ec00836b-iptables-alerter-script\") pod \"iptables-alerter-4smn4\" (UID: \"bd26ace6-2a22-4558-b175-ac19ec00836b\") " pod="openshift-network-operator/iptables-alerter-4smn4"
Sep 16 21:22:21.715146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715151    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-systemd\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd1d301e-815d-4471-a53a-b7a3bd452da8-serviceca\") pod \"node-ca-xfzln\" (UID: \"fd1d301e-815d-4471-a53a-b7a3bd452da8\") " pod="openshift-image-registry/node-ca-xfzln"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-device-dir\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-lib-modules\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f48f3a5-577d-426d-8288-75e7a451c50f-hosts-file\") pod \"node-resolver-ftdzk\" (UID: \"0f48f3a5-577d-426d-8288-75e7a451c50f\") " pod="openshift-dns/node-resolver-ftdzk"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-run\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715264    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-device-dir\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-tmp\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f48f3a5-577d-426d-8288-75e7a451c50f-hosts-file\") pod \"node-resolver-ftdzk\" (UID: \"0f48f3a5-577d-426d-8288-75e7a451c50f\") " pod="openshift-dns/node-resolver-ftdzk"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-registration-dir\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-run\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nspsq\" (UniqueName: \"kubernetes.io/projected/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-kube-api-access-nspsq\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-registration-dir\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/371857b4-159c-47eb-897c-e71bb61469c7-konnectivity-ca\") pod \"konnectivity-agent-s89hp\" (UID: \"371857b4-159c-47eb-897c-e71bb61469c7\") " pod="kube-system/konnectivity-agent-s89hp"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-etc-selinux\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-var-lib-kubelet\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-multus-conf-dir\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.715913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715452    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-sysctl-d\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715466    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715482    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd1d301e-815d-4471-a53a-b7a3bd452da8-host\") pod \"node-ca-xfzln\" (UID: \"fd1d301e-815d-4471-a53a-b7a3bd452da8\") " pod="openshift-image-registry/node-ca-xfzln"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-system-cni-dir\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715517    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-multus-cni-dir\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-hostroot\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/371857b4-159c-47eb-897c-e71bb61469c7-agent-certs\") pod \"konnectivity-agent-s89hp\" (UID: \"371857b4-159c-47eb-897c-e71bb61469c7\") " pod="kube-system/konnectivity-agent-s89hp"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-var-lib-kubelet\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-socket-dir\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-etc-selinux\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhn2p\" (UniqueName: \"kubernetes.io/projected/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-kube-api-access-hhn2p\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd1d301e-815d-4471-a53a-b7a3bd452da8-serviceca\") pod \"node-ca-xfzln\" (UID: \"fd1d301e-815d-4471-a53a-b7a3bd452da8\") " pod="openshift-image-registry/node-ca-xfzln"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-var-lib-kubelet\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-sys-fs\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715718    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-sys-fs\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd1d301e-815d-4471-a53a-b7a3bd452da8-host\") pod \"node-ca-xfzln\" (UID: \"fd1d301e-815d-4471-a53a-b7a3bd452da8\") " pod="openshift-image-registry/node-ca-xfzln"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-multus-socket-dir-parent\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.716698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-system-cni-dir\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715778    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd26ace6-2a22-4558-b175-ac19ec00836b-iptables-alerter-script\") pod \"iptables-alerter-4smn4\" (UID: \"bd26ace6-2a22-4558-b175-ac19ec00836b\") " pod="openshift-network-operator/iptables-alerter-4smn4"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-run-netns\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715823    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-multus-socket-dir-parent\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f48f3a5-577d-426d-8288-75e7a451c50f-tmp-dir\") pod \"node-resolver-ftdzk\" (UID: \"0f48f3a5-577d-426d-8288-75e7a451c50f\") " pod="openshift-dns/node-resolver-ftdzk"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-var-lib-kubelet\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhpnz\" (UniqueName: \"kubernetes.io/projected/c56686e1-b7c9-4db3-9013-28e89905285e-kube-api-access-nhpnz\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-cnibin\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-multus-conf-dir\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c56686e1-b7c9-4db3-9013-28e89905285e-socket-dir\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-cni-binary-copy\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-sysctl-d\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-var-lib-cni-multus\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-etc-kubernetes\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.716024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-multus-cni-dir\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.716030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-run-netns\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.716038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-host\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-hostroot\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.716072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-host-var-lib-cni-multus\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.716085    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-cnibin\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.716088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-etc-kubernetes\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.715902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/371857b4-159c-47eb-897c-e71bb61469c7-konnectivity-ca\") pod \"konnectivity-agent-s89hp\" (UID: \"371857b4-159c-47eb-897c-e71bb61469c7\") " pod="kube-system/konnectivity-agent-s89hp"
Sep 16 21:22:21.717750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.716113    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-multus-daemon-config\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.716118    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-host\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.717750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.716143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-modprobe-d\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.717750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.716167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-sysconfig\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.717750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.716174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f48f3a5-577d-426d-8288-75e7a451c50f-tmp-dir\") pod \"node-resolver-ftdzk\" (UID: \"0f48f3a5-577d-426d-8288-75e7a451c50f\") " pod="openshift-dns/node-resolver-ftdzk"
Sep 16 21:22:21.717750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.716235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-sysconfig\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.717750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.716296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-modprobe-d\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.717750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.716454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-cni-binary-copy\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.717750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.716499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-multus-daemon-config\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.718228 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.717842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-tmp\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.718228 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.718109    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-etc-tuned\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.718228 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.718144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/371857b4-159c-47eb-897c-e71bb61469c7-agent-certs\") pod \"konnectivity-agent-s89hp\" (UID: \"371857b4-159c-47eb-897c-e71bb61469c7\") " pod="kube-system/konnectivity-agent-s89hp"
Sep 16 21:22:21.724256 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.724239    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhn2p\" (UniqueName: \"kubernetes.io/projected/a0721a6e-ab7d-4fc9-b5d7-186e30bee273-kube-api-access-hhn2p\") pod \"multus-7m8bq\" (UID: \"a0721a6e-ab7d-4fc9-b5d7-186e30bee273\") " pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.727140 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.727113    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbrk8\" (UniqueName: \"kubernetes.io/projected/fd1d301e-815d-4471-a53a-b7a3bd452da8-kube-api-access-kbrk8\") pod \"node-ca-xfzln\" (UID: \"fd1d301e-815d-4471-a53a-b7a3bd452da8\") " pod="openshift-image-registry/node-ca-xfzln"
Sep 16 21:22:21.727255 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.727237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbmcf\" (UniqueName: \"kubernetes.io/projected/bd26ace6-2a22-4558-b175-ac19ec00836b-kube-api-access-rbmcf\") pod \"iptables-alerter-4smn4\" (UID: \"bd26ace6-2a22-4558-b175-ac19ec00836b\") " pod="openshift-network-operator/iptables-alerter-4smn4"
Sep 16 21:22:21.727712 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.727692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhpnz\" (UniqueName: \"kubernetes.io/projected/c56686e1-b7c9-4db3-9013-28e89905285e-kube-api-access-nhpnz\") pod \"aws-ebs-csi-driver-node-66xc5\" (UID: \"c56686e1-b7c9-4db3-9013-28e89905285e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.728640 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.728624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nspsq\" (UniqueName: \"kubernetes.io/projected/7bf6e905-2d4a-466e-a34c-c9bd11c6c501-kube-api-access-nspsq\") pod \"tuned-zxv2h\" (UID: \"7bf6e905-2d4a-466e-a34c-c9bd11c6c501\") " pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.728776 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.728761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldmfc\" (UniqueName: \"kubernetes.io/projected/0f48f3a5-577d-426d-8288-75e7a451c50f-kube-api-access-ldmfc\") pod \"node-resolver-ftdzk\" (UID: \"0f48f3a5-577d-426d-8288-75e7a451c50f\") " pod="openshift-dns/node-resolver-ftdzk"
Sep 16 21:22:21.810456 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.810428    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:22:21.818185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.818166    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:21.828345 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:21.828319    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955b6bc5_4971_4cf9_bb64_01e237d331ae.slice/crio-61d2849fc7afd9df5d30545cf06cadba663166c40f1b33e111dc3692547c8381 WatchSource:0}: Error finding container 61d2849fc7afd9df5d30545cf06cadba663166c40f1b33e111dc3692547c8381: Status 404 returned error can't find the container with id 61d2849fc7afd9df5d30545cf06cadba663166c40f1b33e111dc3692547c8381
Sep 16 21:22:21.846876 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.846853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vrl4w"
Sep 16 21:22:21.856001 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:21.855977    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839fa7f2_7940_4702_8a23_d273fa25149e.slice/crio-d8269f8da8f286e13861d715e6b7f69acc6e4d17c85e5e99bba6fa70ca6ecd19 WatchSource:0}: Error finding container d8269f8da8f286e13861d715e6b7f69acc6e4d17c85e5e99bba6fa70ca6ecd19: Status 404 returned error can't find the container with id d8269f8da8f286e13861d715e6b7f69acc6e4d17c85e5e99bba6fa70ca6ecd19
Sep 16 21:22:21.867530 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.867510    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4smn4"
Sep 16 21:22:21.874977 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:21.874957    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd26ace6_2a22_4558_b175_ac19ec00836b.slice/crio-f815360e18e3e9fbb0a61f7e0dfb20623c3182250991b8c64cfcfdcc1e99a052 WatchSource:0}: Error finding container f815360e18e3e9fbb0a61f7e0dfb20623c3182250991b8c64cfcfdcc1e99a052: Status 404 returned error can't find the container with id f815360e18e3e9fbb0a61f7e0dfb20623c3182250991b8c64cfcfdcc1e99a052
Sep 16 21:22:21.876326 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.876307    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s89hp"
Sep 16 21:22:21.892180 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.892054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5"
Sep 16 21:22:21.898225 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:21.898201    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371857b4_159c_47eb_897c_e71bb61469c7.slice/crio-348236cc7af0a0d524c7ce74cdf0a428d0d08287bc64ab44b7456e3a424bc75d WatchSource:0}: Error finding container 348236cc7af0a0d524c7ce74cdf0a428d0d08287bc64ab44b7456e3a424bc75d: Status 404 returned error can't find the container with id 348236cc7af0a0d524c7ce74cdf0a428d0d08287bc64ab44b7456e3a424bc75d
Sep 16 21:22:21.898941 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:21.898925    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196354e2e4c46cf552a666282e669d68.slice/crio-add12486246ffbdd4f699673862e04e2c0d34fce18e1e68acdf50e279bf394e6 WatchSource:0}: Error finding container add12486246ffbdd4f699673862e04e2c0d34fce18e1e68acdf50e279bf394e6: Status 404 returned error can't find the container with id add12486246ffbdd4f699673862e04e2c0d34fce18e1e68acdf50e279bf394e6
Sep 16 21:22:21.900909 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:21.900888    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56686e1_b7c9_4db3_9013_28e89905285e.slice/crio-8a7ceeea284bb6ca717beab439c65d7be01137d65419445c16e91bdc3c88bb17 WatchSource:0}: Error finding container 8a7ceeea284bb6ca717beab439c65d7be01137d65419445c16e91bdc3c88bb17: Status 404 returned error can't find the container with id 8a7ceeea284bb6ca717beab439c65d7be01137d65419445c16e91bdc3c88bb17
Sep 16 21:22:21.912324 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.912307    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ftdzk"
Sep 16 21:22:21.921225 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:21.921203    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f48f3a5_577d_426d_8288_75e7a451c50f.slice/crio-fb22069f23e32a7dbbad3fca310470ff3964c0f26e6924997e110ed9d439dd83 WatchSource:0}: Error finding container fb22069f23e32a7dbbad3fca310470ff3964c0f26e6924997e110ed9d439dd83: Status 404 returned error can't find the container with id fb22069f23e32a7dbbad3fca310470ff3964c0f26e6924997e110ed9d439dd83
Sep 16 21:22:21.935177 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.935161    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zxv2h"
Sep 16 21:22:21.942833 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:21.942805    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf6e905_2d4a_466e_a34c_c9bd11c6c501.slice/crio-9dcf8d76f7d7a30db50fbe804e3390f1c73ef5d37cc637a91bcf7c82a0cac1e1 WatchSource:0}: Error finding container 9dcf8d76f7d7a30db50fbe804e3390f1c73ef5d37cc637a91bcf7c82a0cac1e1: Status 404 returned error can't find the container with id 9dcf8d76f7d7a30db50fbe804e3390f1c73ef5d37cc637a91bcf7c82a0cac1e1
Sep 16 21:22:21.951969 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.951947    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xfzln"
Sep 16 21:22:21.956541 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:21.956522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7m8bq"
Sep 16 21:22:21.960899 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:21.960869    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1d301e_815d_4471_a53a_b7a3bd452da8.slice/crio-03d0f710402b5e2f203f7dc7d466ce48da44ce24ad76388ce4961a8378152640 WatchSource:0}: Error finding container 03d0f710402b5e2f203f7dc7d466ce48da44ce24ad76388ce4961a8378152640: Status 404 returned error can't find the container with id 03d0f710402b5e2f203f7dc7d466ce48da44ce24ad76388ce4961a8378152640
Sep 16 21:22:21.966429 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:22:21.966405    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0721a6e_ab7d_4fc9_b5d7_186e30bee273.slice/crio-ebadb8f8a651015c80827a6bfc8e215cc0a456997784fe9b73ab9b623723100b WatchSource:0}: Error finding container ebadb8f8a651015c80827a6bfc8e215cc0a456997784fe9b73ab9b623723100b: Status 404 returned error can't find the container with id ebadb8f8a651015c80827a6bfc8e215cc0a456997784fe9b73ab9b623723100b
Sep 16 21:22:22.119896 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.119814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:22.119896 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.119889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs\") pod \"network-metrics-daemon-4qxdd\" (UID: \"4254080d-20cf-49fc-a3ab-6ffbf08e4f44\") " pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:22.120123 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:22.119967    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:22.120123 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:22.119973    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:22.120123 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:22.120036    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret podName:8800530f-c408-44d0-a430-bd6d8cd8d95c nodeName:}" failed. No retries permitted until 2026-09-16 21:22:23.120017233 +0000 UTC m=+3.020227235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret") pod "global-pull-secret-syncer-zxtz8" (UID: "8800530f-c408-44d0-a430-bd6d8cd8d95c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:22.120123 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:22.120077    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs podName:4254080d-20cf-49fc-a3ab-6ffbf08e4f44 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:23.120064264 +0000 UTC m=+3.020274246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs") pod "network-metrics-daemon-4qxdd" (UID: "4254080d-20cf-49fc-a3ab-6ffbf08e4f44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:22.220504 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.220446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvn7w\" (UniqueName: \"kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w\") pod \"network-check-target-9pdvd\" (UID: \"b6ce6503-957e-48b0-836a-699b9dd71a86\") " pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:22.220713 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:22.220672    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:22.220713 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:22.220696    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:22.220713 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:22.220708    2563 projected.go:196] Error preparing data for projected volume kube-api-access-gvn7w for pod openshift-network-diagnostics/network-check-target-9pdvd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:22.220863 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:22.220769    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w podName:b6ce6503-957e-48b0-836a-699b9dd71a86 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:23.220750024 +0000 UTC m=+3.120960023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvn7w" (UniqueName: "kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w") pod "network-check-target-9pdvd" (UID: "b6ce6503-957e-48b0-836a-699b9dd71a86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:22.426994 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.426964    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:22:22.585313 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.585248    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 21:17:21 +0000 UTC" deadline="2028-03-09 23:00:54.977345061 +0000 UTC"
Sep 16 21:22:22.585313 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.585311    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12961h38m32.39203834s"
Sep 16 21:22:22.608473 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.608444    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:22.608640 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:22.608565    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:22.616145 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.616123    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:22:22.624847 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.624792    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s89hp" event={"ID":"371857b4-159c-47eb-897c-e71bb61469c7","Type":"ContainerStarted","Data":"348236cc7af0a0d524c7ce74cdf0a428d0d08287bc64ab44b7456e3a424bc75d"}
Sep 16 21:22:22.630759 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.630729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4smn4" event={"ID":"bd26ace6-2a22-4558-b175-ac19ec00836b","Type":"ContainerStarted","Data":"f815360e18e3e9fbb0a61f7e0dfb20623c3182250991b8c64cfcfdcc1e99a052"}
Sep 16 21:22:22.642456 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.640703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz" event={"ID":"955b6bc5-4971-4cf9-bb64-01e237d331ae","Type":"ContainerStarted","Data":"61d2849fc7afd9df5d30545cf06cadba663166c40f1b33e111dc3692547c8381"}
Sep 16 21:22:22.643549 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.643478    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-118.ec2.internal" event={"ID":"126583346e4bbf13f4d978c46ad24fd3","Type":"ContainerStarted","Data":"d4bfc5b682a1244e32c2c4939b1b074e00e084b78c7008beb160f3ca9798f101"}
Sep 16 21:22:22.647634 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.647561    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zxv2h" event={"ID":"7bf6e905-2d4a-466e-a34c-c9bd11c6c501","Type":"ContainerStarted","Data":"9dcf8d76f7d7a30db50fbe804e3390f1c73ef5d37cc637a91bcf7c82a0cac1e1"}
Sep 16 21:22:22.650498 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.650252    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal" event={"ID":"196354e2e4c46cf552a666282e669d68","Type":"ContainerStarted","Data":"add12486246ffbdd4f699673862e04e2c0d34fce18e1e68acdf50e279bf394e6"}
Sep 16 21:22:22.660380 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.660354    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrl4w" event={"ID":"839fa7f2-7940-4702-8a23-d273fa25149e","Type":"ContainerStarted","Data":"d8269f8da8f286e13861d715e6b7f69acc6e4d17c85e5e99bba6fa70ca6ecd19"}
Sep 16 21:22:22.671490 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.671453    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7m8bq" event={"ID":"a0721a6e-ab7d-4fc9-b5d7-186e30bee273","Type":"ContainerStarted","Data":"ebadb8f8a651015c80827a6bfc8e215cc0a456997784fe9b73ab9b623723100b"}
Sep 16 21:22:22.682542 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.682480    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xfzln" event={"ID":"fd1d301e-815d-4471-a53a-b7a3bd452da8","Type":"ContainerStarted","Data":"03d0f710402b5e2f203f7dc7d466ce48da44ce24ad76388ce4961a8378152640"}
Sep 16 21:22:22.689277 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.689243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ftdzk" event={"ID":"0f48f3a5-577d-426d-8288-75e7a451c50f","Type":"ContainerStarted","Data":"fb22069f23e32a7dbbad3fca310470ff3964c0f26e6924997e110ed9d439dd83"}
Sep 16 21:22:22.692104 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:22.692078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5" event={"ID":"c56686e1-b7c9-4db3-9013-28e89905285e","Type":"ContainerStarted","Data":"8a7ceeea284bb6ca717beab439c65d7be01137d65419445c16e91bdc3c88bb17"}
Sep 16 21:22:23.126868 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:23.126686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs\") pod \"network-metrics-daemon-4qxdd\" (UID: \"4254080d-20cf-49fc-a3ab-6ffbf08e4f44\") " pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:23.126868 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:23.126768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:23.127085 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:23.126894    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:23.127085 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:23.126960    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret podName:8800530f-c408-44d0-a430-bd6d8cd8d95c nodeName:}" failed. No retries permitted until 2026-09-16 21:22:25.126942494 +0000 UTC m=+5.027152478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret") pod "global-pull-secret-syncer-zxtz8" (UID: "8800530f-c408-44d0-a430-bd6d8cd8d95c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:23.127565 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:23.127357    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:23.127565 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:23.127409    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs podName:4254080d-20cf-49fc-a3ab-6ffbf08e4f44 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:25.127394222 +0000 UTC m=+5.027604212 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs") pod "network-metrics-daemon-4qxdd" (UID: "4254080d-20cf-49fc-a3ab-6ffbf08e4f44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:23.227476 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:23.227437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvn7w\" (UniqueName: \"kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w\") pod \"network-check-target-9pdvd\" (UID: \"b6ce6503-957e-48b0-836a-699b9dd71a86\") " pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:23.227672 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:23.227640    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:23.227736 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:23.227676    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:23.227736 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:23.227689    2563 projected.go:196] Error preparing data for projected volume kube-api-access-gvn7w for pod openshift-network-diagnostics/network-check-target-9pdvd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:23.227854 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:23.227748    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w podName:b6ce6503-957e-48b0-836a-699b9dd71a86 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:25.227728786 +0000 UTC m=+5.127938768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvn7w" (UniqueName: "kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w") pod "network-check-target-9pdvd" (UID: "b6ce6503-957e-48b0-836a-699b9dd71a86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:23.250430 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:23.250401    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:22:23.586337 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:23.586292    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 21:17:21 +0000 UTC" deadline="2028-04-25 14:10:14.679269739 +0000 UTC"
Sep 16 21:22:23.586337 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:23.586335    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14080h47m51.092939097s"
Sep 16 21:22:23.602803 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:23.602740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:23.602966 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:23.602880    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:23.603307 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:23.603280    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:23.603405 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:23.603385    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:24.603807 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:24.603778    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:24.604370 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:24.603901    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:25.145507 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:25.145467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:25.145743 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:25.145546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs\") pod \"network-metrics-daemon-4qxdd\" (UID: \"4254080d-20cf-49fc-a3ab-6ffbf08e4f44\") " pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:25.145743 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:25.145702    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:25.145886 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:25.145765    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs podName:4254080d-20cf-49fc-a3ab-6ffbf08e4f44 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:29.145744995 +0000 UTC m=+9.045954979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs") pod "network-metrics-daemon-4qxdd" (UID: "4254080d-20cf-49fc-a3ab-6ffbf08e4f44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:25.146164 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:25.146141    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:25.146241 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:25.146204    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret podName:8800530f-c408-44d0-a430-bd6d8cd8d95c nodeName:}" failed. No retries permitted until 2026-09-16 21:22:29.146186845 +0000 UTC m=+9.046396830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret") pod "global-pull-secret-syncer-zxtz8" (UID: "8800530f-c408-44d0-a430-bd6d8cd8d95c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:25.246723 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:25.246597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvn7w\" (UniqueName: \"kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w\") pod \"network-check-target-9pdvd\" (UID: \"b6ce6503-957e-48b0-836a-699b9dd71a86\") " pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:25.246899 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:25.246818    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:25.246899 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:25.246843    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:25.246899 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:25.246870    2563 projected.go:196] Error preparing data for projected volume kube-api-access-gvn7w for pod openshift-network-diagnostics/network-check-target-9pdvd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:25.247263 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:25.247243    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w podName:b6ce6503-957e-48b0-836a-699b9dd71a86 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:29.247219115 +0000 UTC m=+9.147429101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvn7w" (UniqueName: "kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w") pod "network-check-target-9pdvd" (UID: "b6ce6503-957e-48b0-836a-699b9dd71a86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:25.602816 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:25.602774    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:25.603013 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:25.602823    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:25.603013 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:25.602929    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:25.603129 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:25.603060    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:26.604787 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:26.604538    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:26.605239 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:26.604895    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:27.602217 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:27.602181    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:27.602396 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:27.602196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:27.602396 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:27.602302    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:27.602396 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:27.602362    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:28.602767 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:28.602276    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:28.602767 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:28.602403    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:29.179808 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:29.179769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:29.180008 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:29.179854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs\") pod \"network-metrics-daemon-4qxdd\" (UID: \"4254080d-20cf-49fc-a3ab-6ffbf08e4f44\") " pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:29.180008 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:29.179934    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:29.180008 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:29.179970    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:29.180173 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:29.180017    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret podName:8800530f-c408-44d0-a430-bd6d8cd8d95c nodeName:}" failed. No retries permitted until 2026-09-16 21:22:37.17999728 +0000 UTC m=+17.080207266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret") pod "global-pull-secret-syncer-zxtz8" (UID: "8800530f-c408-44d0-a430-bd6d8cd8d95c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:29.180173 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:29.180040    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs podName:4254080d-20cf-49fc-a3ab-6ffbf08e4f44 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:37.180029222 +0000 UTC m=+17.080239209 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs") pod "network-metrics-daemon-4qxdd" (UID: "4254080d-20cf-49fc-a3ab-6ffbf08e4f44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:29.281041 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:29.281006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvn7w\" (UniqueName: \"kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w\") pod \"network-check-target-9pdvd\" (UID: \"b6ce6503-957e-48b0-836a-699b9dd71a86\") " pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:29.281247 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:29.281220    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:29.281247 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:29.281250    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:29.281434 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:29.281264    2563 projected.go:196] Error preparing data for projected volume kube-api-access-gvn7w for pod openshift-network-diagnostics/network-check-target-9pdvd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:29.281434 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:29.281336    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w podName:b6ce6503-957e-48b0-836a-699b9dd71a86 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:37.281313752 +0000 UTC m=+17.181523757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvn7w" (UniqueName: "kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w") pod "network-check-target-9pdvd" (UID: "b6ce6503-957e-48b0-836a-699b9dd71a86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:29.602025 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:29.601939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:29.602025 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:29.602010    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:29.602225 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:29.602137    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:29.602822 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:29.602783    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:30.603619 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:30.603156    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:30.603619 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:30.603272    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:31.602991 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:31.602948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:31.603170 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:31.602948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:31.603170 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:31.603086    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:31.603170 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:31.603153    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:32.602920 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:32.602883    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:32.603349 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:32.603021    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:33.602213 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:33.602171    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:33.602213 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:33.602200    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:33.602415 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:33.602320    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:33.602509 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:33.602486    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:34.602992 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:34.602961    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:34.603408 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:34.603079    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:35.602846 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:35.602810    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:35.603038 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:35.602817    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:35.603038 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:35.602959    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:35.603038 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:35.603026    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:36.602764 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:36.602731    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:36.602961 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:36.602843    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:37.243264 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:37.243230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:37.243723 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:37.243288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs\") pod \"network-metrics-daemon-4qxdd\" (UID: \"4254080d-20cf-49fc-a3ab-6ffbf08e4f44\") " pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:37.243723 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:37.243383    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:37.243723 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:37.243389    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:37.243723 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:37.243437    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs podName:4254080d-20cf-49fc-a3ab-6ffbf08e4f44 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:53.243421253 +0000 UTC m=+33.143631257 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs") pod "network-metrics-daemon-4qxdd" (UID: "4254080d-20cf-49fc-a3ab-6ffbf08e4f44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:37.243723 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:37.243452    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret podName:8800530f-c408-44d0-a430-bd6d8cd8d95c nodeName:}" failed. No retries permitted until 2026-09-16 21:22:53.243444991 +0000 UTC m=+33.143654972 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret") pod "global-pull-secret-syncer-zxtz8" (UID: "8800530f-c408-44d0-a430-bd6d8cd8d95c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:37.343965 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:37.343887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvn7w\" (UniqueName: \"kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w\") pod \"network-check-target-9pdvd\" (UID: \"b6ce6503-957e-48b0-836a-699b9dd71a86\") " pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:37.344111 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:37.344060    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:37.344111 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:37.344098    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:37.344111 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:37.344108    2563 projected.go:196] Error preparing data for projected volume kube-api-access-gvn7w for pod openshift-network-diagnostics/network-check-target-9pdvd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:37.344233 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:37.344163    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w podName:b6ce6503-957e-48b0-836a-699b9dd71a86 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:53.344149359 +0000 UTC m=+33.244359345 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvn7w" (UniqueName: "kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w") pod "network-check-target-9pdvd" (UID: "b6ce6503-957e-48b0-836a-699b9dd71a86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:37.602795 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:37.602717    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:37.602795 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:37.602741    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:37.603029 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:37.602850    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:37.603029 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:37.602931    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:38.602621 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:38.602582    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:38.603099 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:38.602733    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:39.602590 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:39.602555    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:39.602816 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:39.602692    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:39.602816 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:39.602741    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:39.603162 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:39.602833    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:40.603224 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:40.603191    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:40.603555 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:40.603307    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:41.602695 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.602502    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:41.602805 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.602557    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:41.602805 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:41.602785    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:41.602871 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:41.602836    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:41.735607 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.735570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s89hp" event={"ID":"371857b4-159c-47eb-897c-e71bb61469c7","Type":"ContainerStarted","Data":"b859fbbc5ac6492655076850e6070a833a3c209611cb49120235abb87dba69f4"}
Sep 16 21:22:41.739232 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.739198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz" event={"ID":"955b6bc5-4971-4cf9-bb64-01e237d331ae","Type":"ContainerStarted","Data":"17d6a68c5597fa08f1b5da6981191f7c3913cf97edb3ee7614e8d39e4cb7365f"}
Sep 16 21:22:41.739381 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.739271    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz" event={"ID":"955b6bc5-4971-4cf9-bb64-01e237d331ae","Type":"ContainerStarted","Data":"0ce4b434544e26d4a45a7cc5ff89524e90a98a6cf5d6b31a578417afbb5a8c19"}
Sep 16 21:22:41.739381 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.739315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz" event={"ID":"955b6bc5-4971-4cf9-bb64-01e237d331ae","Type":"ContainerStarted","Data":"bfef3f8a6bb3ac41db92b631941b8643b7e87dd31c1f5a8ec209bbda22f64d3d"}
Sep 16 21:22:41.739381 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.739345    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz" event={"ID":"955b6bc5-4971-4cf9-bb64-01e237d331ae","Type":"ContainerStarted","Data":"ad05cf45e3277dca2b659156f5562ed457aac4cbd4e7a5f863421723eb0a7906"}
Sep 16 21:22:41.739534 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.739381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz" event={"ID":"955b6bc5-4971-4cf9-bb64-01e237d331ae","Type":"ContainerStarted","Data":"debeada61ae644a8e33fbacee6cea5696f8ff19e87a93542ab2a2ba1be3fa00b"}
Sep 16 21:22:41.739534 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.739415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz" event={"ID":"955b6bc5-4971-4cf9-bb64-01e237d331ae","Type":"ContainerStarted","Data":"57251fe795aa31a98bdcdf766305cfb875de6659ceaa7ee3628818111285defc"}
Sep 16 21:22:41.740890 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.740863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-118.ec2.internal" event={"ID":"126583346e4bbf13f4d978c46ad24fd3","Type":"ContainerStarted","Data":"4346ce1f9edc878f26edc5a2d2ecd8ff69ddb39e8df5fbb0618ae9ac439bf6be"}
Sep 16 21:22:41.742473 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.742448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zxv2h" event={"ID":"7bf6e905-2d4a-466e-a34c-c9bd11c6c501","Type":"ContainerStarted","Data":"9195f87bc6c473633e9db3e320873235885845113a9e0614f7375a438c9a05d3"}
Sep 16 21:22:41.744132 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.744105    2563 generic.go:356] "Generic (PLEG): container finished" podID="196354e2e4c46cf552a666282e669d68" containerID="05a256873c365464fbb0ca5d819aff3ce0d8d4e41811495718a248e207990cab" exitCode=0
Sep 16 21:22:41.744202 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.744167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal" event={"ID":"196354e2e4c46cf552a666282e669d68","Type":"ContainerDied","Data":"05a256873c365464fbb0ca5d819aff3ce0d8d4e41811495718a248e207990cab"}
Sep 16 21:22:41.745940 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.745918    2563 generic.go:356] "Generic (PLEG): container finished" podID="839fa7f2-7940-4702-8a23-d273fa25149e" containerID="352bda7e2aaaf6f96531493d0322a96ed8468d2426c9f9351888bea97def7568" exitCode=0
Sep 16 21:22:41.746036 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.746007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrl4w" event={"ID":"839fa7f2-7940-4702-8a23-d273fa25149e","Type":"ContainerDied","Data":"352bda7e2aaaf6f96531493d0322a96ed8468d2426c9f9351888bea97def7568"}
Sep 16 21:22:41.747664 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.747615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7m8bq" event={"ID":"a0721a6e-ab7d-4fc9-b5d7-186e30bee273","Type":"ContainerStarted","Data":"b1ccd0ea48680f232039f05f411c2f346825f91b0df7737ce570474efa114155"}
Sep 16 21:22:41.749635 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.749614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xfzln" event={"ID":"fd1d301e-815d-4471-a53a-b7a3bd452da8","Type":"ContainerStarted","Data":"64641d1b7cc2d13c37434f6c556c6fb7ec48eeb05fd925c90596fa44a763ca11"}
Sep 16 21:22:41.750494 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.750452    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s89hp" podStartSLOduration=2.824061834 podStartE2EDuration="21.750439463s" totalImagesPullingTime="18.926377629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:41.750221681 +0000 UTC m=+21.650431686" watchObservedRunningTime="2026-09-16 21:22:41.750439463 +0000 UTC m=+21.650649466"
Sep 16 21:22:41.751317 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.751292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ftdzk" event={"ID":"0f48f3a5-577d-426d-8288-75e7a451c50f","Type":"ContainerStarted","Data":"e0830c162742df5b048efc4562fee021304440eda96e5293a0891b79f48044ba"}
Sep 16 21:22:41.752785 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.752765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5" event={"ID":"c56686e1-b7c9-4db3-9013-28e89905285e","Type":"ContainerStarted","Data":"824777ef64affd1801297a31039cfccc05609a94d9bb367c14e12abb4e527762"}
Sep 16 21:22:41.763129 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.763085    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-118.ec2.internal" podStartSLOduration=20.76307146 podStartE2EDuration="20.76307146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:41.7627476 +0000 UTC m=+21.662957604" watchObservedRunningTime="2026-09-16 21:22:41.76307146 +0000 UTC m=+21.663281464"
Sep 16 21:22:41.780283 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.780238    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zxv2h" podStartSLOduration=2.865994744 podStartE2EDuration="21.780223952s" totalImagesPullingTime="18.914229208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:41.779879921 +0000 UTC m=+21.680089925" watchObservedRunningTime="2026-09-16 21:22:41.780223952 +0000 UTC m=+21.680433957"
Sep 16 21:22:41.800784 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.800737    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7m8bq" podStartSLOduration=2.754478876 podStartE2EDuration="21.800722121s" totalImagesPullingTime="19.046243245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:41.800429539 +0000 UTC m=+21.700639544" watchObservedRunningTime="2026-09-16 21:22:41.800722121 +0000 UTC m=+21.700932124"
Sep 16 21:22:41.840950 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.840903    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xfzln" podStartSLOduration=2.994066153 podStartE2EDuration="21.840885331s" totalImagesPullingTime="18.846819178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:41.840311844 +0000 UTC m=+21.740521847" watchObservedRunningTime="2026-09-16 21:22:41.840885331 +0000 UTC m=+21.741095326"
Sep 16 21:22:41.856672 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:41.856611    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ftdzk" podStartSLOduration=2.919595471 podStartE2EDuration="21.856595486s" totalImagesPullingTime="18.937000015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:41.856557291 +0000 UTC m=+21.756767295" watchObservedRunningTime="2026-09-16 21:22:41.856595486 +0000 UTC m=+21.756805490"
Sep 16 21:22:42.602330 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:42.602240    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:42.602491 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:42.602371    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:42.757412 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:42.757335    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4smn4" event={"ID":"bd26ace6-2a22-4558-b175-ac19ec00836b","Type":"ContainerStarted","Data":"7f8fbda08a1c2485056bd57a0c6c0e00305b32c61c6940258e9b1d3260b09994"}
Sep 16 21:22:42.771005 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:42.770965    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4smn4" podStartSLOduration=3.820925181 podStartE2EDuration="22.770948802s" totalImagesPullingTime="18.950023621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:42.770800038 +0000 UTC m=+22.671010043" watchObservedRunningTime="2026-09-16 21:22:42.770948802 +0000 UTC m=+22.671158800"
Sep 16 21:22:42.944945 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:42.944918    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 21:22:43.569799 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:43.569681    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T21:22:42.944940894Z","UUID":"b594993e-d5cd-444c-b8ab-94cc7d03508e","Handler":null,"Name":"","Endpoint":""}
Sep 16 21:22:43.571549 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:43.571524    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 21:22:43.571719 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:43.571564    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 21:22:43.602061 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:43.602019    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:43.602263 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:43.602019    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:43.602263 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:43.602124    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:43.602263 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:43.602219    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:43.763228 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:43.762678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5" event={"ID":"c56686e1-b7c9-4db3-9013-28e89905285e","Type":"ContainerStarted","Data":"a6f2c759c8c3fa33f6d7e6cf1d0efc0aebb8372bbd07492fc85db93dac43c06c"}
Sep 16 21:22:43.765740 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:43.765698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal" event={"ID":"196354e2e4c46cf552a666282e669d68","Type":"ContainerStarted","Data":"132cb53c4958251bb95c0661667ae7fa12ce75e688fef8f209cce1646327422a"}
Sep 16 21:22:43.782492 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:43.782431    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-118.ec2.internal" podStartSLOduration=22.782408262 podStartE2EDuration="22.782408262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:43.781794827 +0000 UTC m=+23.682004843" watchObservedRunningTime="2026-09-16 21:22:43.782408262 +0000 UTC m=+23.682618269"
Sep 16 21:22:44.602976 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:44.602901    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:44.603156 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:44.603020    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:44.770441 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:44.770249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz" event={"ID":"955b6bc5-4971-4cf9-bb64-01e237d331ae","Type":"ContainerStarted","Data":"ae3306688218ef8f1facafa9b8113fdff37274236678bb4dd6e8d04815b95497"}
Sep 16 21:22:44.772293 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:44.772269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5" event={"ID":"c56686e1-b7c9-4db3-9013-28e89905285e","Type":"ContainerStarted","Data":"5fbbe75c90e90904895823f5dd4858c54b158187b250d8aa8f35eb827e983bdd"}
Sep 16 21:22:44.790116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:44.790065    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66xc5" podStartSLOduration=3.162263837 podStartE2EDuration="24.790051989s" totalImagesPullingTime="21.627788152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:44.789835602 +0000 UTC m=+24.690045606" watchObservedRunningTime="2026-09-16 21:22:44.790051989 +0000 UTC m=+24.690261993"
Sep 16 21:22:45.602054 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:45.602018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:45.602264 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:45.602018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:45.602264 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:45.602161    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:45.602264 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:45.602215    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:46.413018 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:46.412929    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s89hp"
Sep 16 21:22:46.413584 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:46.413565    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s89hp"
Sep 16 21:22:46.602341 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:46.602198    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:46.602341 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:46.602300    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:46.781067 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:46.781029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz" event={"ID":"955b6bc5-4971-4cf9-bb64-01e237d331ae","Type":"ContainerStarted","Data":"054d9354814f11e6fe500ac2a2248cd1ac846dc614ffce9312732a662464977f"}
Sep 16 21:22:46.781436 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:46.781396    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:46.783565 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:46.783539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrl4w" event={"ID":"839fa7f2-7940-4702-8a23-d273fa25149e","Type":"ContainerStarted","Data":"ebcc6dbf40e35879d52c782c466ee26958d04d9bb4b21be19d7b658d0ce21e43"}
Sep 16 21:22:46.797974 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:46.797948    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:46.807963 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:46.807785    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz" podStartSLOduration=7.744033095 podStartE2EDuration="26.807764601s" totalImagesPullingTime="19.063731506s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:46.806586696 +0000 UTC m=+26.706796692" watchObservedRunningTime="2026-09-16 21:22:46.807764601 +0000 UTC m=+26.707974607"
Sep 16 21:22:47.602424 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:47.602385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:47.602424 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:47.602425    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:47.602925 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:47.602519    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:47.602925 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:47.602636    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:47.787281 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:47.787250    2563 generic.go:356] "Generic (PLEG): container finished" podID="839fa7f2-7940-4702-8a23-d273fa25149e" containerID="ebcc6dbf40e35879d52c782c466ee26958d04d9bb4b21be19d7b658d0ce21e43" exitCode=0
Sep 16 21:22:47.787428 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:47.787339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrl4w" event={"ID":"839fa7f2-7940-4702-8a23-d273fa25149e","Type":"ContainerDied","Data":"ebcc6dbf40e35879d52c782c466ee26958d04d9bb4b21be19d7b658d0ce21e43"}
Sep 16 21:22:47.787428 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:47.787371    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 21:22:47.787824 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:47.787798    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:47.802705 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:47.802677    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:48.602704 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:48.602678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:48.602982 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:48.602778    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:48.791612 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:48.791521    2563 generic.go:356] "Generic (PLEG): container finished" podID="839fa7f2-7940-4702-8a23-d273fa25149e" containerID="93db6abd5ac4ccfe91b3fbbe6a3a54a2e56dcf20845314901d838e6bb5014d50" exitCode=0
Sep 16 21:22:48.791783 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:48.791605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrl4w" event={"ID":"839fa7f2-7940-4702-8a23-d273fa25149e","Type":"ContainerDied","Data":"93db6abd5ac4ccfe91b3fbbe6a3a54a2e56dcf20845314901d838e6bb5014d50"}
Sep 16 21:22:48.791783 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:48.791675    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 21:22:49.602062 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:49.601968    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:49.602062 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:49.602010    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:49.602229 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:49.602079    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:49.602229 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:49.602195    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:49.769834 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:49.769803    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:22:49.796199 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:49.796170    2563 generic.go:356] "Generic (PLEG): container finished" podID="839fa7f2-7940-4702-8a23-d273fa25149e" containerID="d57e774301c08bb8b127c79abf56abd5efa4ca249be7a8466429a0ef9d192aed" exitCode=0
Sep 16 21:22:49.796324 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:49.796252    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrl4w" event={"ID":"839fa7f2-7940-4702-8a23-d273fa25149e","Type":"ContainerDied","Data":"d57e774301c08bb8b127c79abf56abd5efa4ca249be7a8466429a0ef9d192aed"}
Sep 16 21:22:50.537176 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:50.537142    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s89hp"
Sep 16 21:22:50.537350 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:50.537267    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 21:22:50.537685 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:50.537671    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s89hp"
Sep 16 21:22:50.603197 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:50.603164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:50.603336 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:50.603274    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:51.602994 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:51.602956    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:51.603561 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:51.602971    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:51.603561 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:51.603084    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:51.603561 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:51.603159    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:52.602777 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:52.602721    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:52.602950 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:52.602878    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:53.255129 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:53.255095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs\") pod \"network-metrics-daemon-4qxdd\" (UID: \"4254080d-20cf-49fc-a3ab-6ffbf08e4f44\") " pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:53.255493 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:53.255150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:53.255493 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:53.255237    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:53.255493 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:53.255238    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:53.255493 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:53.255290    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret podName:8800530f-c408-44d0-a430-bd6d8cd8d95c nodeName:}" failed. No retries permitted until 2026-09-16 21:23:25.255277756 +0000 UTC m=+65.155487751 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret") pod "global-pull-secret-syncer-zxtz8" (UID: "8800530f-c408-44d0-a430-bd6d8cd8d95c") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:53.255493 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:53.255305    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs podName:4254080d-20cf-49fc-a3ab-6ffbf08e4f44 nodeName:}" failed. No retries permitted until 2026-09-16 21:23:25.255298519 +0000 UTC m=+65.155508502 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs") pod "network-metrics-daemon-4qxdd" (UID: "4254080d-20cf-49fc-a3ab-6ffbf08e4f44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:53.355440 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:53.355395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvn7w\" (UniqueName: \"kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w\") pod \"network-check-target-9pdvd\" (UID: \"b6ce6503-957e-48b0-836a-699b9dd71a86\") " pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:53.355600 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:53.355582    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:53.355672 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:53.355607    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:53.355672 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:53.355620    2563 projected.go:196] Error preparing data for projected volume kube-api-access-gvn7w for pod openshift-network-diagnostics/network-check-target-9pdvd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:53.355753 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:53.355734    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w podName:b6ce6503-957e-48b0-836a-699b9dd71a86 nodeName:}" failed. No retries permitted until 2026-09-16 21:23:25.355713867 +0000 UTC m=+65.255923855 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvn7w" (UniqueName: "kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w") pod "network-check-target-9pdvd" (UID: "b6ce6503-957e-48b0-836a-699b9dd71a86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:53.601983 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:53.601887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:53.602147 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:53.601898    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:53.602147 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:53.602037    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:53.602147 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:53.602078    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:54.602762 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:54.602724    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:54.603310 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:54.602858    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:55.602111 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:55.602084    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:55.602111 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:55.602103    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:55.602252 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:55.602194    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:55.602311 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:55.602294    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:55.814064 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:55.813991    2563 generic.go:356] "Generic (PLEG): container finished" podID="839fa7f2-7940-4702-8a23-d273fa25149e" containerID="527ce65c005ad84db7cf9ddafe161aff316d181ed1d03d6e5b28272b6abda24b" exitCode=0
Sep 16 21:22:55.814064 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:55.814054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrl4w" event={"ID":"839fa7f2-7940-4702-8a23-d273fa25149e","Type":"ContainerDied","Data":"527ce65c005ad84db7cf9ddafe161aff316d181ed1d03d6e5b28272b6abda24b"}
Sep 16 21:22:56.602322 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:56.602287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:56.602498 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:56.602397    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:56.819372 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:56.819338    2563 generic.go:356] "Generic (PLEG): container finished" podID="839fa7f2-7940-4702-8a23-d273fa25149e" containerID="b6b3fa0c7ffc261d552de3e37bf01d0fdb9ef065ad2b1b80e82a5f84b43a95a5" exitCode=0
Sep 16 21:22:56.819756 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:56.819379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrl4w" event={"ID":"839fa7f2-7940-4702-8a23-d273fa25149e","Type":"ContainerDied","Data":"b6b3fa0c7ffc261d552de3e37bf01d0fdb9ef065ad2b1b80e82a5f84b43a95a5"}
Sep 16 21:22:57.602159 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:57.602126    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:57.602334 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:57.602235    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:57.602334 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:57.602291    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:57.602410 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:57.602374    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:22:57.824961 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:57.824919    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vrl4w" event={"ID":"839fa7f2-7940-4702-8a23-d273fa25149e","Type":"ContainerStarted","Data":"e8db6b0d346da0e2bf7c7afbfa0d168fd74c651bc58935a475b731880187a4e4"}
Sep 16 21:22:57.849170 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:57.849120    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vrl4w" podStartSLOduration=6.939662263 podStartE2EDuration="37.849104532s" totalImagesPullingTime="30.909442269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:57.848168824 +0000 UTC m=+37.748378827" watchObservedRunningTime="2026-09-16 21:22:57.849104532 +0000 UTC m=+37.749314536"
Sep 16 21:22:58.601948 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:58.601916    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:22:58.602137 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:58.602034    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:22:59.602458 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:59.602424    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:22:59.602858 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:59.602529    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:22:59.602858 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:22:59.602584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:22:59.602858 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:22:59.602676    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:23:00.605766 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:00.605738    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:23:00.606259 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:00.605832    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:23:01.602274 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:01.602238    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:23:01.602274 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:01.602265    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:23:01.602486 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:01.602364    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:23:01.602486 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:01.602474    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:23:02.605358 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:02.605322    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:23:02.605737 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:02.605425    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:23:03.602872 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:03.602830    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:23:03.603086 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:03.602829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:23:03.603086 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:03.602970    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:23:03.603086 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:03.603014    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:23:04.602504 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:04.602468    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:23:04.602912 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:04.602574    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:23:05.602135 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:05.601949    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:23:05.602271 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:05.602000    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:23:05.602271 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:05.602211    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:23:05.602339 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:05.602274    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:23:05.796680 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:05.796597    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zxtz8"]
Sep 16 21:23:05.801041 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:05.801012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4qxdd"]
Sep 16 21:23:05.801900 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:05.801876    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9pdvd"]
Sep 16 21:23:05.802045 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:05.801969    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:23:05.802103 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:05.802062    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:23:05.843314 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:05.843213    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:23:05.843314 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:05.843214    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:23:05.843516 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:05.843348    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:23:05.843516 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:05.843380    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:23:07.602956 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:07.602923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:23:07.603396 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:07.602923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:23:07.603396 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:07.603037    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:23:07.603396 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:07.602923    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:23:07.603396 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:07.603113    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:23:07.603396 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:07.603215    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:23:09.602272 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.602230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:23:09.602272 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.602230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:23:09.602604 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.602236    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:23:09.602604 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:09.602352    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9pdvd" podUID="b6ce6503-957e-48b0-836a-699b9dd71a86"
Sep 16 21:23:09.602604 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:09.602454    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4qxdd" podUID="4254080d-20cf-49fc-a3ab-6ffbf08e4f44"
Sep 16 21:23:09.602604 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:09.602526    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zxtz8" podUID="8800530f-c408-44d0-a430-bd6d8cd8d95c"
Sep 16 21:23:09.862456 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.862428    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-118.ec2.internal" event="NodeReady"
Sep 16 21:23:09.862613 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.862555    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 21:23:09.896759 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.896731    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-765979c47f-j8hjs"]
Sep 16 21:23:09.916172 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.916124    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:09.917761 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.917731    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-765979c47f-j8hjs"]
Sep 16 21:23:09.918409 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.918390    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vlfj5"]
Sep 16 21:23:09.919061 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.918767    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 21:23:09.919061 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.918781    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 21:23:09.919061 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.918837    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5496m\""
Sep 16 21:23:09.919061 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.918877    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 21:23:09.925215 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.924992    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 21:23:09.930531 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.930512    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-khxgl"]
Sep 16 21:23:09.930633 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.930617    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:09.934415 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.934395    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rcnph\""
Sep 16 21:23:09.934682 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.934649    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 21:23:09.935014 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.934995    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 21:23:09.952467 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.952440    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-khxgl"]
Sep 16 21:23:09.952586 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.952483    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vlfj5"]
Sep 16 21:23:09.952586 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.952551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:09.956332 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.956191    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 21:23:09.956332 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.956212    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 21:23:09.956332 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.956236    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 21:23:09.956332 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.956259    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hr64t\""
Sep 16 21:23:09.956332 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.956274    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 21:23:09.967344 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.967324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl4gz\" (UniqueName: \"kubernetes.io/projected/b084665b-6abf-4507-8dad-41d4af410da4-kube-api-access-sl4gz\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:09.967459 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.967391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b084665b-6abf-4507-8dad-41d4af410da4-image-registry-private-configuration\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:09.967459 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.967419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b084665b-6abf-4507-8dad-41d4af410da4-ca-trust-extracted\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:09.967565 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.967461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b084665b-6abf-4507-8dad-41d4af410da4-registry-tls\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:09.967565 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.967483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b084665b-6abf-4507-8dad-41d4af410da4-bound-sa-token\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:09.967565 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.967511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b084665b-6abf-4507-8dad-41d4af410da4-registry-certificates\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:09.967565 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.967538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b084665b-6abf-4507-8dad-41d4af410da4-installation-pull-secrets\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:09.967738 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:09.967574    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b084665b-6abf-4507-8dad-41d4af410da4-trusted-ca\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.016793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.016760    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vtbqv"]
Sep 16 21:23:10.039179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.039154    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vtbqv"]
Sep 16 21:23:10.039317 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.039247    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vtbqv"
Sep 16 21:23:10.043415 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.043397    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 21:23:10.043537 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.043519    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 21:23:10.043586 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.043541    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 21:23:10.043586 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.043573    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-77lfh\""
Sep 16 21:23:10.068640 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.068568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7hgj\" (UniqueName: \"kubernetes.io/projected/2dde1c15-7080-4d1e-afc3-6b91a8555c10-kube-api-access-s7hgj\") pod \"dns-default-vlfj5\" (UID: \"2dde1c15-7080-4d1e-afc3-6b91a8555c10\") " pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:10.068640 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.068597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/efdae95d-7f63-4795-998a-ed27ce7b8131-crio-socket\") pod \"insights-runtime-extractor-khxgl\" (UID: \"efdae95d-7f63-4795-998a-ed27ce7b8131\") " pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.068640 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.068619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b084665b-6abf-4507-8dad-41d4af410da4-image-registry-private-configuration\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.068861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.068684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b084665b-6abf-4507-8dad-41d4af410da4-ca-trust-extracted\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.068861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.068719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/efdae95d-7f63-4795-998a-ed27ce7b8131-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khxgl\" (UID: \"efdae95d-7f63-4795-998a-ed27ce7b8131\") " pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.068861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.068773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b084665b-6abf-4507-8dad-41d4af410da4-registry-tls\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.068861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.068820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b084665b-6abf-4507-8dad-41d4af410da4-bound-sa-token\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.068861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.068848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/efdae95d-7f63-4795-998a-ed27ce7b8131-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khxgl\" (UID: \"efdae95d-7f63-4795-998a-ed27ce7b8131\") " pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.069066 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.068880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b084665b-6abf-4507-8dad-41d4af410da4-registry-certificates\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.069066 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.068903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2dde1c15-7080-4d1e-afc3-6b91a8555c10-config-volume\") pod \"dns-default-vlfj5\" (UID: \"2dde1c15-7080-4d1e-afc3-6b91a8555c10\") " pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:10.069066 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.068925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/efdae95d-7f63-4795-998a-ed27ce7b8131-data-volume\") pod \"insights-runtime-extractor-khxgl\" (UID: \"efdae95d-7f63-4795-998a-ed27ce7b8131\") " pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.069066 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.068952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b084665b-6abf-4507-8dad-41d4af410da4-installation-pull-secrets\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.069066 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.069006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b084665b-6abf-4507-8dad-41d4af410da4-trusted-ca\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.069272 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.069078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2dde1c15-7080-4d1e-afc3-6b91a8555c10-metrics-tls\") pod \"dns-default-vlfj5\" (UID: \"2dde1c15-7080-4d1e-afc3-6b91a8555c10\") " pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:10.069272 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.069118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s46ng\" (UniqueName: \"kubernetes.io/projected/efdae95d-7f63-4795-998a-ed27ce7b8131-kube-api-access-s46ng\") pod \"insights-runtime-extractor-khxgl\" (UID: \"efdae95d-7f63-4795-998a-ed27ce7b8131\") " pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.069272 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.069157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl4gz\" (UniqueName: \"kubernetes.io/projected/b084665b-6abf-4507-8dad-41d4af410da4-kube-api-access-sl4gz\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.069272 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.069189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dde1c15-7080-4d1e-afc3-6b91a8555c10-tmp-dir\") pod \"dns-default-vlfj5\" (UID: \"2dde1c15-7080-4d1e-afc3-6b91a8555c10\") " pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:10.069272 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.069209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b084665b-6abf-4507-8dad-41d4af410da4-ca-trust-extracted\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.070095 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.070042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b084665b-6abf-4507-8dad-41d4af410da4-trusted-ca\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.070095 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.070078    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b084665b-6abf-4507-8dad-41d4af410da4-registry-certificates\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.072974 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.072867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b084665b-6abf-4507-8dad-41d4af410da4-registry-tls\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.072974 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.072885    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b084665b-6abf-4507-8dad-41d4af410da4-installation-pull-secrets\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.073100 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.072894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b084665b-6abf-4507-8dad-41d4af410da4-image-registry-private-configuration\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.084702 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.084675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b084665b-6abf-4507-8dad-41d4af410da4-bound-sa-token\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.084828 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.084814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl4gz\" (UniqueName: \"kubernetes.io/projected/b084665b-6abf-4507-8dad-41d4af410da4-kube-api-access-sl4gz\") pod \"image-registry-765979c47f-j8hjs\" (UID: \"b084665b-6abf-4507-8dad-41d4af410da4\") " pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.169898 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.169847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x2f4\" (UniqueName: \"kubernetes.io/projected/6e8d3c7a-f1d2-4bcf-a3dd-7898c531d61b-kube-api-access-7x2f4\") pod \"ingress-canary-vtbqv\" (UID: \"6e8d3c7a-f1d2-4bcf-a3dd-7898c531d61b\") " pod="openshift-ingress-canary/ingress-canary-vtbqv"
Sep 16 21:23:10.170046 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.169917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2dde1c15-7080-4d1e-afc3-6b91a8555c10-metrics-tls\") pod \"dns-default-vlfj5\" (UID: \"2dde1c15-7080-4d1e-afc3-6b91a8555c10\") " pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:10.170046 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.169941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s46ng\" (UniqueName: \"kubernetes.io/projected/efdae95d-7f63-4795-998a-ed27ce7b8131-kube-api-access-s46ng\") pod \"insights-runtime-extractor-khxgl\" (UID: \"efdae95d-7f63-4795-998a-ed27ce7b8131\") " pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.170046 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.169961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dde1c15-7080-4d1e-afc3-6b91a8555c10-tmp-dir\") pod \"dns-default-vlfj5\" (UID: \"2dde1c15-7080-4d1e-afc3-6b91a8555c10\") " pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:10.170046 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.169977    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e8d3c7a-f1d2-4bcf-a3dd-7898c531d61b-cert\") pod \"ingress-canary-vtbqv\" (UID: \"6e8d3c7a-f1d2-4bcf-a3dd-7898c531d61b\") " pod="openshift-ingress-canary/ingress-canary-vtbqv"
Sep 16 21:23:10.170046 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.170032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7hgj\" (UniqueName: \"kubernetes.io/projected/2dde1c15-7080-4d1e-afc3-6b91a8555c10-kube-api-access-s7hgj\") pod \"dns-default-vlfj5\" (UID: \"2dde1c15-7080-4d1e-afc3-6b91a8555c10\") " pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:10.170283 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.170058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/efdae95d-7f63-4795-998a-ed27ce7b8131-crio-socket\") pod \"insights-runtime-extractor-khxgl\" (UID: \"efdae95d-7f63-4795-998a-ed27ce7b8131\") " pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.170283 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.170086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/efdae95d-7f63-4795-998a-ed27ce7b8131-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khxgl\" (UID: \"efdae95d-7f63-4795-998a-ed27ce7b8131\") " pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.170283 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.170126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/efdae95d-7f63-4795-998a-ed27ce7b8131-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khxgl\" (UID: \"efdae95d-7f63-4795-998a-ed27ce7b8131\") " pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.170283 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.170149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2dde1c15-7080-4d1e-afc3-6b91a8555c10-config-volume\") pod \"dns-default-vlfj5\" (UID: \"2dde1c15-7080-4d1e-afc3-6b91a8555c10\") " pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:10.170283 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.170175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/efdae95d-7f63-4795-998a-ed27ce7b8131-data-volume\") pod \"insights-runtime-extractor-khxgl\" (UID: \"efdae95d-7f63-4795-998a-ed27ce7b8131\") " pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.170489 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.170353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dde1c15-7080-4d1e-afc3-6b91a8555c10-tmp-dir\") pod \"dns-default-vlfj5\" (UID: \"2dde1c15-7080-4d1e-afc3-6b91a8555c10\") " pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:10.170489 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.170451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/efdae95d-7f63-4795-998a-ed27ce7b8131-crio-socket\") pod \"insights-runtime-extractor-khxgl\" (UID: \"efdae95d-7f63-4795-998a-ed27ce7b8131\") " pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.170629 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.170595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/efdae95d-7f63-4795-998a-ed27ce7b8131-data-volume\") pod \"insights-runtime-extractor-khxgl\" (UID: \"efdae95d-7f63-4795-998a-ed27ce7b8131\") " pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.170783 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.170760    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/efdae95d-7f63-4795-998a-ed27ce7b8131-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khxgl\" (UID: \"efdae95d-7f63-4795-998a-ed27ce7b8131\") " pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.170855 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.170806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2dde1c15-7080-4d1e-afc3-6b91a8555c10-config-volume\") pod \"dns-default-vlfj5\" (UID: \"2dde1c15-7080-4d1e-afc3-6b91a8555c10\") " pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:10.172524 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.172497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2dde1c15-7080-4d1e-afc3-6b91a8555c10-metrics-tls\") pod \"dns-default-vlfj5\" (UID: \"2dde1c15-7080-4d1e-afc3-6b91a8555c10\") " pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:10.172524 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.172511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/efdae95d-7f63-4795-998a-ed27ce7b8131-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khxgl\" (UID: \"efdae95d-7f63-4795-998a-ed27ce7b8131\") " pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.193815 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.193781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s46ng\" (UniqueName: \"kubernetes.io/projected/efdae95d-7f63-4795-998a-ed27ce7b8131-kube-api-access-s46ng\") pod \"insights-runtime-extractor-khxgl\" (UID: \"efdae95d-7f63-4795-998a-ed27ce7b8131\") " pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.193944 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.193879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7hgj\" (UniqueName: \"kubernetes.io/projected/2dde1c15-7080-4d1e-afc3-6b91a8555c10-kube-api-access-s7hgj\") pod \"dns-default-vlfj5\" (UID: \"2dde1c15-7080-4d1e-afc3-6b91a8555c10\") " pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:10.227840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.227808    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.240037 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.240007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:10.264168 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.264137    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-khxgl"
Sep 16 21:23:10.271545 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.271520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e8d3c7a-f1d2-4bcf-a3dd-7898c531d61b-cert\") pod \"ingress-canary-vtbqv\" (UID: \"6e8d3c7a-f1d2-4bcf-a3dd-7898c531d61b\") " pod="openshift-ingress-canary/ingress-canary-vtbqv"
Sep 16 21:23:10.271662 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.271641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x2f4\" (UniqueName: \"kubernetes.io/projected/6e8d3c7a-f1d2-4bcf-a3dd-7898c531d61b-kube-api-access-7x2f4\") pod \"ingress-canary-vtbqv\" (UID: \"6e8d3c7a-f1d2-4bcf-a3dd-7898c531d61b\") " pod="openshift-ingress-canary/ingress-canary-vtbqv"
Sep 16 21:23:10.274063 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.274046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e8d3c7a-f1d2-4bcf-a3dd-7898c531d61b-cert\") pod \"ingress-canary-vtbqv\" (UID: \"6e8d3c7a-f1d2-4bcf-a3dd-7898c531d61b\") " pod="openshift-ingress-canary/ingress-canary-vtbqv"
Sep 16 21:23:10.282735 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.282715    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x2f4\" (UniqueName: \"kubernetes.io/projected/6e8d3c7a-f1d2-4bcf-a3dd-7898c531d61b-kube-api-access-7x2f4\") pod \"ingress-canary-vtbqv\" (UID: \"6e8d3c7a-f1d2-4bcf-a3dd-7898c531d61b\") " pod="openshift-ingress-canary/ingress-canary-vtbqv"
Sep 16 21:23:10.347722 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.347636    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vtbqv"
Sep 16 21:23:10.561456 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.561422    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-765979c47f-j8hjs"]
Sep 16 21:23:10.566623 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:23:10.566593    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb084665b_6abf_4507_8dad_41d4af410da4.slice/crio-b6fd2dbdc0a97fa73c940ea190c08b210f41d90b2837db729e63c7b12d9a1a14 WatchSource:0}: Error finding container b6fd2dbdc0a97fa73c940ea190c08b210f41d90b2837db729e63c7b12d9a1a14: Status 404 returned error can't find the container with id b6fd2dbdc0a97fa73c940ea190c08b210f41d90b2837db729e63c7b12d9a1a14
Sep 16 21:23:10.574968 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.574916    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vlfj5"]
Sep 16 21:23:10.577243 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:23:10.577222    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dde1c15_7080_4d1e_afc3_6b91a8555c10.slice/crio-cfcb998e4608d31fcd1662f89f975fa5148d28fe10f22a12919a99b1e0a91ab4 WatchSource:0}: Error finding container cfcb998e4608d31fcd1662f89f975fa5148d28fe10f22a12919a99b1e0a91ab4: Status 404 returned error can't find the container with id cfcb998e4608d31fcd1662f89f975fa5148d28fe10f22a12919a99b1e0a91ab4
Sep 16 21:23:10.608940 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.608907    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-khxgl"]
Sep 16 21:23:10.616782 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:23:10.616755    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdae95d_7f63_4795_998a_ed27ce7b8131.slice/crio-eafb3a269992f370738d88011b7fee82481a59112bfab96836f4f0aefca1d3fc WatchSource:0}: Error finding container eafb3a269992f370738d88011b7fee82481a59112bfab96836f4f0aefca1d3fc: Status 404 returned error can't find the container with id eafb3a269992f370738d88011b7fee82481a59112bfab96836f4f0aefca1d3fc
Sep 16 21:23:10.642906 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.642884    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vtbqv"]
Sep 16 21:23:10.646617 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:23:10.646597    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8d3c7a_f1d2_4bcf_a3dd_7898c531d61b.slice/crio-2663c7d935220d336f60c3be73f93d3144e18e0e997c1ff50edbe0c92b2c55b4 WatchSource:0}: Error finding container 2663c7d935220d336f60c3be73f93d3144e18e0e997c1ff50edbe0c92b2c55b4: Status 404 returned error can't find the container with id 2663c7d935220d336f60c3be73f93d3144e18e0e997c1ff50edbe0c92b2c55b4
Sep 16 21:23:10.854810 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.854719    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-765979c47f-j8hjs" event={"ID":"b084665b-6abf-4507-8dad-41d4af410da4","Type":"ContainerStarted","Data":"2c05653c635ea882c5b8f24b99d964e8e12097896e67201744ab7a6beea8b1a7"}
Sep 16 21:23:10.854810 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.854799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-765979c47f-j8hjs" event={"ID":"b084665b-6abf-4507-8dad-41d4af410da4","Type":"ContainerStarted","Data":"b6fd2dbdc0a97fa73c940ea190c08b210f41d90b2837db729e63c7b12d9a1a14"}
Sep 16 21:23:10.855064 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.854828    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:10.856050 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.856026    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vtbqv" event={"ID":"6e8d3c7a-f1d2-4bcf-a3dd-7898c531d61b","Type":"ContainerStarted","Data":"2663c7d935220d336f60c3be73f93d3144e18e0e997c1ff50edbe0c92b2c55b4"}
Sep 16 21:23:10.857585 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.857559    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-khxgl" event={"ID":"efdae95d-7f63-4795-998a-ed27ce7b8131","Type":"ContainerStarted","Data":"1bdd11882ea6bd518447c598c492cfd20bf1f9e90b92aa1db7e281c3383be7a4"}
Sep 16 21:23:10.857694 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.857620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-khxgl" event={"ID":"efdae95d-7f63-4795-998a-ed27ce7b8131","Type":"ContainerStarted","Data":"eafb3a269992f370738d88011b7fee82481a59112bfab96836f4f0aefca1d3fc"}
Sep 16 21:23:10.859216 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.859188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vlfj5" event={"ID":"2dde1c15-7080-4d1e-afc3-6b91a8555c10","Type":"ContainerStarted","Data":"cfcb998e4608d31fcd1662f89f975fa5148d28fe10f22a12919a99b1e0a91ab4"}
Sep 16 21:23:10.895025 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:10.894968    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-765979c47f-j8hjs" podStartSLOduration=6.894952132 podStartE2EDuration="6.894952132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:10.894587918 +0000 UTC m=+50.794797921" watchObservedRunningTime="2026-09-16 21:23:10.894952132 +0000 UTC m=+50.795162132"
Sep 16 21:23:11.603125 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:11.602878    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:23:11.603309 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:11.602947    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:23:11.603309 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:11.602959    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:23:11.609019 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:11.608730    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 21:23:11.609019 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:11.608842    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hdj86\""
Sep 16 21:23:11.611370 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:11.608731    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 21:23:11.611370 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:11.609131    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 21:23:11.611370 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:11.609274    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 21:23:11.611370 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:11.609969    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8xpmt\""
Sep 16 21:23:11.864750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:11.864667    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-khxgl" event={"ID":"efdae95d-7f63-4795-998a-ed27ce7b8131","Type":"ContainerStarted","Data":"1705290e815704697b9bf63670216588cb0720780bb6305269cac8462849e0c8"}
Sep 16 21:23:14.874640 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:14.874605    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vlfj5" event={"ID":"2dde1c15-7080-4d1e-afc3-6b91a8555c10","Type":"ContainerStarted","Data":"ec59a4490ca7f27990f7d3cdaf28d5a26a62e089cef1a608608eb3598c02ad24"}
Sep 16 21:23:14.875162 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:14.874721    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:14.875162 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:14.874752    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vlfj5" event={"ID":"2dde1c15-7080-4d1e-afc3-6b91a8555c10","Type":"ContainerStarted","Data":"88c6b48f70de08c1c9598417f37a60c743896cb6ad7f6969eb9c10a16902906a"}
Sep 16 21:23:14.876191 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:14.876164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vtbqv" event={"ID":"6e8d3c7a-f1d2-4bcf-a3dd-7898c531d61b","Type":"ContainerStarted","Data":"7fc7caf11290a54de9ff4ce8384ed1d28662b1da8cc9eeada27dde6b485b5649"}
Sep 16 21:23:14.878184 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:14.878157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-khxgl" event={"ID":"efdae95d-7f63-4795-998a-ed27ce7b8131","Type":"ContainerStarted","Data":"c774e1435bb536152937627e328523271879bc9d2d7b95dd547aa4e69c9d201f"}
Sep 16 21:23:14.891214 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:14.891051    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vlfj5" podStartSLOduration=2.569067884 podStartE2EDuration="5.891036285s" totalImagesPullingTime="3.321968401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:14.890539323 +0000 UTC m=+54.790749326" watchObservedRunningTime="2026-09-16 21:23:14.891036285 +0000 UTC m=+54.791246291"
Sep 16 21:23:14.907441 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:14.907391    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-khxgl" podStartSLOduration=2.779184475 podStartE2EDuration="5.907372689s" totalImagesPullingTime="3.128188214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:23:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:14.906733947 +0000 UTC m=+54.806943952" watchObservedRunningTime="2026-09-16 21:23:14.907372689 +0000 UTC m=+54.807582693"
Sep 16 21:23:14.922177 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:14.922127    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vtbqv" podStartSLOduration=2.672204974 podStartE2EDuration="5.922110974s" totalImagesPullingTime="3.249906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:14.921588874 +0000 UTC m=+54.821798906" watchObservedRunningTime="2026-09-16 21:23:14.922110974 +0000 UTC m=+54.822320979"
Sep 16 21:23:15.099624 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.099586    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5756969dff-h294g"]
Sep 16 21:23:15.102579 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.102561    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.104767 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.104743    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 21:23:15.104892 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.104753    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 21:23:15.104892 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.104818    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 21:23:15.104892 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.104830    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 21:23:15.105046 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.104997    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 21:23:15.105046 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.105033    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 21:23:15.105142 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.105054    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 21:23:15.105244 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.105228    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vpfpz\""
Sep 16 21:23:15.110292 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.110268    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 21:23:15.114063 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.114042    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5756969dff-h294g"]
Sep 16 21:23:15.216712 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.216685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tqlt\" (UniqueName: \"kubernetes.io/projected/17fae843-8922-4f91-8c20-9e1abc4161b8-kube-api-access-9tqlt\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.216833 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.216724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-oauth-serving-cert\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.216833 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.216754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-trusted-ca-bundle\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.216959 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.216832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-service-ca\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.216959 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.216873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17fae843-8922-4f91-8c20-9e1abc4161b8-console-serving-cert\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.216959 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.216891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17fae843-8922-4f91-8c20-9e1abc4161b8-console-oauth-config\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.216959 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.216907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-console-config\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.317218 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.317183    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tqlt\" (UniqueName: \"kubernetes.io/projected/17fae843-8922-4f91-8c20-9e1abc4161b8-kube-api-access-9tqlt\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.317218 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.317219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-oauth-serving-cert\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.317429 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.317243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-trusted-ca-bundle\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.317429 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.317302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-service-ca\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.317429 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.317345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17fae843-8922-4f91-8c20-9e1abc4161b8-console-serving-cert\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.317560 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.317480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17fae843-8922-4f91-8c20-9e1abc4161b8-console-oauth-config\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.317560 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.317518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-console-config\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.318183 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.318131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-service-ca\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.318183 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.318174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-console-config\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.318356 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.318185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-oauth-serving-cert\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.318356 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.318285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-trusted-ca-bundle\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.320452 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.320429    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17fae843-8922-4f91-8c20-9e1abc4161b8-console-serving-cert\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.320555 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.320517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17fae843-8922-4f91-8c20-9e1abc4161b8-console-oauth-config\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.326833 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.326811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tqlt\" (UniqueName: \"kubernetes.io/projected/17fae843-8922-4f91-8c20-9e1abc4161b8-kube-api-access-9tqlt\") pod \"console-5756969dff-h294g\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") " pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.413271 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.413230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:15.689497 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.689461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5756969dff-h294g"]
Sep 16 21:23:15.692582 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:23:15.692551    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fae843_8922_4f91_8c20_9e1abc4161b8.slice/crio-26c557798de6a74eb19a62c46dde85051d3cf1935e9fc8a377dc6395f292e3ec WatchSource:0}: Error finding container 26c557798de6a74eb19a62c46dde85051d3cf1935e9fc8a377dc6395f292e3ec: Status 404 returned error can't find the container with id 26c557798de6a74eb19a62c46dde85051d3cf1935e9fc8a377dc6395f292e3ec
Sep 16 21:23:15.883148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:15.883063    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5756969dff-h294g" event={"ID":"17fae843-8922-4f91-8c20-9e1abc4161b8","Type":"ContainerStarted","Data":"26c557798de6a74eb19a62c46dde85051d3cf1935e9fc8a377dc6395f292e3ec"}
Sep 16 21:23:18.894113 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:18.894080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5756969dff-h294g" event={"ID":"17fae843-8922-4f91-8c20-9e1abc4161b8","Type":"ContainerStarted","Data":"d0fbd8ca966e5516fc105a9ffdd6c49f4708069eaebe0fa348bd3511891eb9f8"}
Sep 16 21:23:18.915725 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:18.915674    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5756969dff-h294g" podStartSLOduration=1.255130242 podStartE2EDuration="3.915639324s" totalImagesPullingTime="2.660509082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:18.914308463 +0000 UTC m=+58.814518466" watchObservedRunningTime="2026-09-16 21:23:18.915639324 +0000 UTC m=+58.815849327"
Sep 16 21:23:20.808931 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:20.808902    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lk5dz"
Sep 16 21:23:24.695170 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.695136    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"]
Sep 16 21:23:24.700831 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.700811    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"
Sep 16 21:23:24.703037 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.703012    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 21:23:24.703190 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.703096    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 21:23:24.703471 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.703454    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 21:23:24.703742 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.703720    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 21:23:24.703808 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.703795    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 21:23:24.703854 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.703818    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-j99nr\""
Sep 16 21:23:24.709751 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.709730    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"]
Sep 16 21:23:24.740137 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.740110    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tqsbg"]
Sep 16 21:23:24.743999 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.743980    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.746205 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.746188    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 21:23:24.746303 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.746255    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 21:23:24.746736 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.746720    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 21:23:24.746793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.746750    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q9ngt\""
Sep 16 21:23:24.795921 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.795887    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/654340c3-d71e-4069-9fff-78a36f7cc28a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-dn5cq\" (UID: \"654340c3-d71e-4069-9fff-78a36f7cc28a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"
Sep 16 21:23:24.796106 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.795926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zmwh\" (UniqueName: \"kubernetes.io/projected/654340c3-d71e-4069-9fff-78a36f7cc28a-kube-api-access-6zmwh\") pod \"openshift-state-metrics-67dcf8c979-dn5cq\" (UID: \"654340c3-d71e-4069-9fff-78a36f7cc28a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"
Sep 16 21:23:24.796106 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.795960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/654340c3-d71e-4069-9fff-78a36f7cc28a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-dn5cq\" (UID: \"654340c3-d71e-4069-9fff-78a36f7cc28a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"
Sep 16 21:23:24.796106 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.796030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/654340c3-d71e-4069-9fff-78a36f7cc28a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-dn5cq\" (UID: \"654340c3-d71e-4069-9fff-78a36f7cc28a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"
Sep 16 21:23:24.885149 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.885119    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vlfj5"
Sep 16 21:23:24.897359 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.897325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/654340c3-d71e-4069-9fff-78a36f7cc28a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-dn5cq\" (UID: \"654340c3-d71e-4069-9fff-78a36f7cc28a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"
Sep 16 21:23:24.897476 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.897373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59972e1c-28dd-4a1e-8163-66defade025a-root\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.897476 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.897390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59972e1c-28dd-4a1e-8163-66defade025a-node-exporter-textfile\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.897476 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.897406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59972e1c-28dd-4a1e-8163-66defade025a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.897476 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.897452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59972e1c-28dd-4a1e-8163-66defade025a-node-exporter-tls\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.897711 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.897491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/654340c3-d71e-4069-9fff-78a36f7cc28a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-dn5cq\" (UID: \"654340c3-d71e-4069-9fff-78a36f7cc28a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"
Sep 16 21:23:24.897711 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:24.897580    2563 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 16 21:23:24.897711 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.897609    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59972e1c-28dd-4a1e-8163-66defade025a-node-exporter-wtmp\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.897711 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:23:24.897638    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/654340c3-d71e-4069-9fff-78a36f7cc28a-openshift-state-metrics-tls podName:654340c3-d71e-4069-9fff-78a36f7cc28a nodeName:}" failed. No retries permitted until 2026-09-16 21:23:25.397619084 +0000 UTC m=+65.297829079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/654340c3-d71e-4069-9fff-78a36f7cc28a-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-dn5cq" (UID: "654340c3-d71e-4069-9fff-78a36f7cc28a") : secret "openshift-state-metrics-tls" not found
Sep 16 21:23:24.897711 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.897689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59972e1c-28dd-4a1e-8163-66defade025a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.897920 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.897729    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59972e1c-28dd-4a1e-8163-66defade025a-metrics-client-ca\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.897920 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.897747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59972e1c-28dd-4a1e-8163-66defade025a-sys\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.897920 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.897762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ktn7\" (UniqueName: \"kubernetes.io/projected/59972e1c-28dd-4a1e-8163-66defade025a-kube-api-access-8ktn7\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.897920 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.897789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/654340c3-d71e-4069-9fff-78a36f7cc28a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-dn5cq\" (UID: \"654340c3-d71e-4069-9fff-78a36f7cc28a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"
Sep 16 21:23:24.897920 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.897826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zmwh\" (UniqueName: \"kubernetes.io/projected/654340c3-d71e-4069-9fff-78a36f7cc28a-kube-api-access-6zmwh\") pod \"openshift-state-metrics-67dcf8c979-dn5cq\" (UID: \"654340c3-d71e-4069-9fff-78a36f7cc28a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"
Sep 16 21:23:24.898159 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.898140    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/654340c3-d71e-4069-9fff-78a36f7cc28a-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-dn5cq\" (UID: \"654340c3-d71e-4069-9fff-78a36f7cc28a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"
Sep 16 21:23:24.900148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.900131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/654340c3-d71e-4069-9fff-78a36f7cc28a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-dn5cq\" (UID: \"654340c3-d71e-4069-9fff-78a36f7cc28a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"
Sep 16 21:23:24.915464 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.915437    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zmwh\" (UniqueName: \"kubernetes.io/projected/654340c3-d71e-4069-9fff-78a36f7cc28a-kube-api-access-6zmwh\") pod \"openshift-state-metrics-67dcf8c979-dn5cq\" (UID: \"654340c3-d71e-4069-9fff-78a36f7cc28a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"
Sep 16 21:23:24.999020 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.998922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59972e1c-28dd-4a1e-8163-66defade025a-node-exporter-tls\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.999020 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.999003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59972e1c-28dd-4a1e-8163-66defade025a-node-exporter-wtmp\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.999232 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.999024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59972e1c-28dd-4a1e-8163-66defade025a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.999232 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.999050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59972e1c-28dd-4a1e-8163-66defade025a-metrics-client-ca\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.999232 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.999069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59972e1c-28dd-4a1e-8163-66defade025a-sys\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.999232 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.999093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ktn7\" (UniqueName: \"kubernetes.io/projected/59972e1c-28dd-4a1e-8163-66defade025a-kube-api-access-8ktn7\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.999232 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.999148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59972e1c-28dd-4a1e-8163-66defade025a-root\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.999232 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.999171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59972e1c-28dd-4a1e-8163-66defade025a-node-exporter-textfile\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.999232 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.999186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59972e1c-28dd-4a1e-8163-66defade025a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.999232 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.999183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59972e1c-28dd-4a1e-8163-66defade025a-node-exporter-wtmp\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.999627 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.999420    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59972e1c-28dd-4a1e-8163-66defade025a-sys\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.999627 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.999438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59972e1c-28dd-4a1e-8163-66defade025a-root\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:24.999780 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:24.999761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59972e1c-28dd-4a1e-8163-66defade025a-node-exporter-textfile\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:25.000068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.000045    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59972e1c-28dd-4a1e-8163-66defade025a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:25.000152 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.000128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59972e1c-28dd-4a1e-8163-66defade025a-metrics-client-ca\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:25.001519 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.001492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59972e1c-28dd-4a1e-8163-66defade025a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:25.001648 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.001562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59972e1c-28dd-4a1e-8163-66defade025a-node-exporter-tls\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:25.007329 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.007306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ktn7\" (UniqueName: \"kubernetes.io/projected/59972e1c-28dd-4a1e-8163-66defade025a-kube-api-access-8ktn7\") pod \"node-exporter-tqsbg\" (UID: \"59972e1c-28dd-4a1e-8163-66defade025a\") " pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:25.056464 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.056429    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqsbg"
Sep 16 21:23:25.068774 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:23:25.068748    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59972e1c_28dd_4a1e_8163_66defade025a.slice/crio-c75395ed2754fcd6bc55fe7045c782dfb0844b391bba491328f29a92f5d847c9 WatchSource:0}: Error finding container c75395ed2754fcd6bc55fe7045c782dfb0844b391bba491328f29a92f5d847c9: Status 404 returned error can't find the container with id c75395ed2754fcd6bc55fe7045c782dfb0844b391bba491328f29a92f5d847c9
Sep 16 21:23:25.301615 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.301519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:23:25.301615 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.301570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs\") pod \"network-metrics-daemon-4qxdd\" (UID: \"4254080d-20cf-49fc-a3ab-6ffbf08e4f44\") " pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:23:25.304681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.304648    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 21:23:25.304870 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.304857    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 21:23:25.314735 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.314706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8800530f-c408-44d0-a430-bd6d8cd8d95c-original-pull-secret\") pod \"global-pull-secret-syncer-zxtz8\" (UID: \"8800530f-c408-44d0-a430-bd6d8cd8d95c\") " pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:23:25.314841 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.314704    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4254080d-20cf-49fc-a3ab-6ffbf08e4f44-metrics-certs\") pod \"network-metrics-daemon-4qxdd\" (UID: \"4254080d-20cf-49fc-a3ab-6ffbf08e4f44\") " pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:23:25.402231 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.402192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/654340c3-d71e-4069-9fff-78a36f7cc28a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-dn5cq\" (UID: \"654340c3-d71e-4069-9fff-78a36f7cc28a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"
Sep 16 21:23:25.402349 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.402243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvn7w\" (UniqueName: \"kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w\") pod \"network-check-target-9pdvd\" (UID: \"b6ce6503-957e-48b0-836a-699b9dd71a86\") " pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:23:25.404540 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.404517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/654340c3-d71e-4069-9fff-78a36f7cc28a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-dn5cq\" (UID: \"654340c3-d71e-4069-9fff-78a36f7cc28a\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"
Sep 16 21:23:25.404668 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.404635    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 21:23:25.413625 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.413604    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:25.413682 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.413670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:25.414464 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.414444    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 21:23:25.418532 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.418515    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:25.420293 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.420275    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8xpmt\""
Sep 16 21:23:25.425460 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.425443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvn7w\" (UniqueName: \"kubernetes.io/projected/b6ce6503-957e-48b0-836a-699b9dd71a86-kube-api-access-gvn7w\") pod \"network-check-target-9pdvd\" (UID: \"b6ce6503-957e-48b0-836a-699b9dd71a86\") " pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:23:25.427216 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.427201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zxtz8"
Sep 16 21:23:25.429302 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.429290    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4qxdd"
Sep 16 21:23:25.436261 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.436241    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hdj86\""
Sep 16 21:23:25.444720 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.444696    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:23:25.609929 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.609854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"
Sep 16 21:23:25.707018 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.706980    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4qxdd"]
Sep 16 21:23:25.710432 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.710330    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zxtz8"]
Sep 16 21:23:25.710620 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:23:25.710592    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4254080d_20cf_49fc_a3ab_6ffbf08e4f44.slice/crio-295a7710837355cbcb432d17829c398dc1f0b3314022e0d753bf9b9205395e8a WatchSource:0}: Error finding container 295a7710837355cbcb432d17829c398dc1f0b3314022e0d753bf9b9205395e8a: Status 404 returned error can't find the container with id 295a7710837355cbcb432d17829c398dc1f0b3314022e0d753bf9b9205395e8a
Sep 16 21:23:25.713435 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:23:25.713399    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8800530f_c408_44d0_a430_bd6d8cd8d95c.slice/crio-9e8966c94716f363aed584a027bc1e8231ed24fd201332dd06a83ab2a66eea5c WatchSource:0}: Error finding container 9e8966c94716f363aed584a027bc1e8231ed24fd201332dd06a83ab2a66eea5c: Status 404 returned error can't find the container with id 9e8966c94716f363aed584a027bc1e8231ed24fd201332dd06a83ab2a66eea5c
Sep 16 21:23:25.729838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.729812    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9pdvd"]
Sep 16 21:23:25.732750 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:23:25.732713    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ce6503_957e_48b0_836a_699b9dd71a86.slice/crio-624534e71a6b47d80384f35ca252bba12e58e700ce5565ac29a13fffdfc2aa2d WatchSource:0}: Error finding container 624534e71a6b47d80384f35ca252bba12e58e700ce5565ac29a13fffdfc2aa2d: Status 404 returned error can't find the container with id 624534e71a6b47d80384f35ca252bba12e58e700ce5565ac29a13fffdfc2aa2d
Sep 16 21:23:25.882699 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.882603    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq"]
Sep 16 21:23:25.885590 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:23:25.885561    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654340c3_d71e_4069_9fff_78a36f7cc28a.slice/crio-224d193f5175983ff2fb6439e60d98ed6a4cb76388bddc27c38feee725190235 WatchSource:0}: Error finding container 224d193f5175983ff2fb6439e60d98ed6a4cb76388bddc27c38feee725190235: Status 404 returned error can't find the container with id 224d193f5175983ff2fb6439e60d98ed6a4cb76388bddc27c38feee725190235
Sep 16 21:23:25.917262 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.917228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9pdvd" event={"ID":"b6ce6503-957e-48b0-836a-699b9dd71a86","Type":"ContainerStarted","Data":"624534e71a6b47d80384f35ca252bba12e58e700ce5565ac29a13fffdfc2aa2d"}
Sep 16 21:23:25.918614 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.918587    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq" event={"ID":"654340c3-d71e-4069-9fff-78a36f7cc28a","Type":"ContainerStarted","Data":"224d193f5175983ff2fb6439e60d98ed6a4cb76388bddc27c38feee725190235"}
Sep 16 21:23:25.919965 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.919915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4qxdd" event={"ID":"4254080d-20cf-49fc-a3ab-6ffbf08e4f44","Type":"ContainerStarted","Data":"295a7710837355cbcb432d17829c398dc1f0b3314022e0d753bf9b9205395e8a"}
Sep 16 21:23:25.921179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.921149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zxtz8" event={"ID":"8800530f-c408-44d0-a430-bd6d8cd8d95c","Type":"ContainerStarted","Data":"9e8966c94716f363aed584a027bc1e8231ed24fd201332dd06a83ab2a66eea5c"}
Sep 16 21:23:25.922451 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.922402    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqsbg" event={"ID":"59972e1c-28dd-4a1e-8163-66defade025a","Type":"ContainerStarted","Data":"c75395ed2754fcd6bc55fe7045c782dfb0844b391bba491328f29a92f5d847c9"}
Sep 16 21:23:25.926864 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:25.926840    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:23:26.929747 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:26.929641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq" event={"ID":"654340c3-d71e-4069-9fff-78a36f7cc28a","Type":"ContainerStarted","Data":"5f94d035d1cf9ba6ac34392d61bf36ee55495f4d649ab5ca15ff999184f9c593"}
Sep 16 21:23:26.929747 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:26.929735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq" event={"ID":"654340c3-d71e-4069-9fff-78a36f7cc28a","Type":"ContainerStarted","Data":"d1e3858ff20b7821c8845abdb3782a93fd69bccb2e3e3b577f04fb900f01a965"}
Sep 16 21:23:26.931823 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:26.931785    2563 generic.go:356] "Generic (PLEG): container finished" podID="59972e1c-28dd-4a1e-8163-66defade025a" containerID="0cf4a9550312846d9ff5f4a3c5c12b33ebdfa601399ac91d3a00bb36bdb4e09d" exitCode=0
Sep 16 21:23:26.931983 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:26.931877    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqsbg" event={"ID":"59972e1c-28dd-4a1e-8163-66defade025a","Type":"ContainerDied","Data":"0cf4a9550312846d9ff5f4a3c5c12b33ebdfa601399ac91d3a00bb36bdb4e09d"}
Sep 16 21:23:27.938336 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:27.938298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq" event={"ID":"654340c3-d71e-4069-9fff-78a36f7cc28a","Type":"ContainerStarted","Data":"6a8b11f71085c14c2085ea813d715427e3d6ed6cadf3e6db25a9561b931a7802"}
Sep 16 21:23:27.940513 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:27.940479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4qxdd" event={"ID":"4254080d-20cf-49fc-a3ab-6ffbf08e4f44","Type":"ContainerStarted","Data":"811913a37b33bfb36cf52e8357f6512dd16039f52e1cc773acdcb3cb7b395d3b"}
Sep 16 21:23:27.940693 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:27.940536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4qxdd" event={"ID":"4254080d-20cf-49fc-a3ab-6ffbf08e4f44","Type":"ContainerStarted","Data":"425a5ba88213c507a5ff98a9a9b293b1435eb6382be22bb5f0cad2cd9e9f196c"}
Sep 16 21:23:27.942952 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:27.942926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqsbg" event={"ID":"59972e1c-28dd-4a1e-8163-66defade025a","Type":"ContainerStarted","Data":"14bc36a8a25e5c590503a70ccab3b6518c175a39bcf070f89d838130fee354d2"}
Sep 16 21:23:27.943060 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:27.942977    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqsbg" event={"ID":"59972e1c-28dd-4a1e-8163-66defade025a","Type":"ContainerStarted","Data":"4eabf5f6c52c5f34671d78c4d5d05f272783da5bb0bcbf6bc58237040602f328"}
Sep 16 21:23:27.959306 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:27.959252    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dn5cq" podStartSLOduration=2.524701847 podStartE2EDuration="3.959235774s" totalImagesPullingTime="1.434533927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:27.958379775 +0000 UTC m=+67.858589780" watchObservedRunningTime="2026-09-16 21:23:27.959235774 +0000 UTC m=+67.859445780"
Sep 16 21:23:27.984463 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:27.984402    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tqsbg" podStartSLOduration=2.830461481 podStartE2EDuration="3.984378382s" totalImagesPullingTime="1.07521224s" totalInitContainerRuntime="78.704661ms" isStatefulPod=true podCreationTimestamp="2026-09-16 21:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:27.983579072 +0000 UTC m=+67.883789074" watchObservedRunningTime="2026-09-16 21:23:27.984378382 +0000 UTC m=+67.884588389"
Sep 16 21:23:28.001671 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:28.001603    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4qxdd" podStartSLOduration=66.856537854 podStartE2EDuration="1m8.001585231s" totalImagesPullingTime="1.145047377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:28.000617164 +0000 UTC m=+67.900827200" watchObservedRunningTime="2026-09-16 21:23:28.001585231 +0000 UTC m=+67.901795236"
Sep 16 21:23:29.782527 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:29.782485    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76ffffc8c9-5s826"]
Sep 16 21:23:29.800791 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:29.800761    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:29.843076 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:29.843041    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76ffffc8c9-5s826"]
Sep 16 21:23:29.944303 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:29.944249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-trusted-ca-bundle\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:29.944303 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:29.944308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-service-ca\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:29.944558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:29.944339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74def189-8a69-4b37-a34a-6302d0978328-console-serving-cert\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:29.944558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:29.944393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74def189-8a69-4b37-a34a-6302d0978328-console-oauth-config\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:29.944558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:29.944424    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-console-config\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:29.944558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:29.944454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-oauth-serving-cert\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:29.944558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:29.944498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkclw\" (UniqueName: \"kubernetes.io/projected/74def189-8a69-4b37-a34a-6302d0978328-kube-api-access-zkclw\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.045108 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.045013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-service-ca\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.045108 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.045065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74def189-8a69-4b37-a34a-6302d0978328-console-serving-cert\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.045108 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.045107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74def189-8a69-4b37-a34a-6302d0978328-console-oauth-config\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.045386 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.045128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-console-config\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.045386 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.045185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-oauth-serving-cert\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.045386 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.045229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkclw\" (UniqueName: \"kubernetes.io/projected/74def189-8a69-4b37-a34a-6302d0978328-kube-api-access-zkclw\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.045386 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.045263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-trusted-ca-bundle\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.045871 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.045845    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-service-ca\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.046010 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.045917    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-console-config\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.046010 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.045995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-oauth-serving-cert\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.046139 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.046056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-trusted-ca-bundle\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.047700 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.047674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74def189-8a69-4b37-a34a-6302d0978328-console-serving-cert\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.047700 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.047685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74def189-8a69-4b37-a34a-6302d0978328-console-oauth-config\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.060385 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.060363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkclw\" (UniqueName: \"kubernetes.io/projected/74def189-8a69-4b37-a34a-6302d0978328-kube-api-access-zkclw\") pod \"console-76ffffc8c9-5s826\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") " pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.111188 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.111143    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:30.114278 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.114244    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-db94579bd-tldhc"]
Sep 16 21:23:30.117560 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.117539    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.126624 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.126598    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 21:23:30.126754 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.126648    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kpq88\""
Sep 16 21:23:30.126805 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.126758    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 21:23:30.126837 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.126598    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 21:23:30.127015 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.126995    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 21:23:30.127131 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.127034    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 21:23:30.132492 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.132467    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 21:23:30.140601 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.140528    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-db94579bd-tldhc"]
Sep 16 21:23:30.247607 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.247566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1793a6c-4e04-461f-b8dd-d557ef233005-metrics-client-ca\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.247799 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.247620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48nzc\" (UniqueName: \"kubernetes.io/projected/a1793a6c-4e04-461f-b8dd-d557ef233005-kube-api-access-48nzc\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.247799 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.247681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1793a6c-4e04-461f-b8dd-d557ef233005-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.247799 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.247726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a1793a6c-4e04-461f-b8dd-d557ef233005-telemeter-client-tls\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.247799 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.247757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1793a6c-4e04-461f-b8dd-d557ef233005-serving-certs-ca-bundle\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.248033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.247818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a1793a6c-4e04-461f-b8dd-d557ef233005-federate-client-tls\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.248033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.247877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1793a6c-4e04-461f-b8dd-d557ef233005-telemeter-trusted-ca-bundle\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.248033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.247959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a1793a6c-4e04-461f-b8dd-d557ef233005-secret-telemeter-client\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.349033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.348947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1793a6c-4e04-461f-b8dd-d557ef233005-metrics-client-ca\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.349033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.349003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48nzc\" (UniqueName: \"kubernetes.io/projected/a1793a6c-4e04-461f-b8dd-d557ef233005-kube-api-access-48nzc\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.349245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.349056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1793a6c-4e04-461f-b8dd-d557ef233005-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.349245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.349100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a1793a6c-4e04-461f-b8dd-d557ef233005-telemeter-client-tls\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.349245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.349134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1793a6c-4e04-461f-b8dd-d557ef233005-serving-certs-ca-bundle\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.349245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.349168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a1793a6c-4e04-461f-b8dd-d557ef233005-federate-client-tls\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.349245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.349192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1793a6c-4e04-461f-b8dd-d557ef233005-telemeter-trusted-ca-bundle\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.349245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.349227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a1793a6c-4e04-461f-b8dd-d557ef233005-secret-telemeter-client\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.350416 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.349818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1793a6c-4e04-461f-b8dd-d557ef233005-metrics-client-ca\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.350416 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.350033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1793a6c-4e04-461f-b8dd-d557ef233005-serving-certs-ca-bundle\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.350416 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.350372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1793a6c-4e04-461f-b8dd-d557ef233005-telemeter-trusted-ca-bundle\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.352154 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.352110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a1793a6c-4e04-461f-b8dd-d557ef233005-secret-telemeter-client\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.352534 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.352510    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a1793a6c-4e04-461f-b8dd-d557ef233005-telemeter-client-tls\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.352629 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.352515    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1793a6c-4e04-461f-b8dd-d557ef233005-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.352629 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.352562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a1793a6c-4e04-461f-b8dd-d557ef233005-federate-client-tls\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.357979 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.357956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48nzc\" (UniqueName: \"kubernetes.io/projected/a1793a6c-4e04-461f-b8dd-d557ef233005-kube-api-access-48nzc\") pod \"telemeter-client-db94579bd-tldhc\" (UID: \"a1793a6c-4e04-461f-b8dd-d557ef233005\") " pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.427897 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.427854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-db94579bd-tldhc"
Sep 16 21:23:30.948736 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.948700    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 21:23:30.952854 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.952822    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:30.955354 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.955328    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 21:23:30.955504 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.955386    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 21:23:30.955730 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.955704    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 21:23:30.955823 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.955774    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 21:23:30.955932 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.955916    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 21:23:30.955974 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.955928    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 21:23:30.955974 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.955960    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 21:23:30.956188 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.956160    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e80ii0cstljva\""
Sep 16 21:23:30.956188 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.956177    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rtjnz\""
Sep 16 21:23:30.956338 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.956208    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 21:23:30.956338 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.956170    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 21:23:30.956575 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.956553    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 21:23:30.956673 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.956605    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 21:23:30.956673 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.956642    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 21:23:30.956779 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.956684    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 21:23:30.956834 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.956800    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 21:23:30.958705 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.958685    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 21:23:30.972594 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:30.972563    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 21:23:31.054948 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.054913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-web-config\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055076 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.054965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055076 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.054997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055076 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055047    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055204 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055204 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055095    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-config-out\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055204 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055303 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055303 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055303 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055287    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055317    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z652\" (UniqueName: \"kubernetes.io/projected/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-kube-api-access-2z652\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055500 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055500 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055447    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-config\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055500 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055630 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055630 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055630 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.055630 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.055555    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-web-config\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-config-out\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z652\" (UniqueName: \"kubernetes.io/projected/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-kube-api-access-2z652\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-config\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158989 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158989 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158989 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.156823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.158989 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.157649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.163928 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.163897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.164215 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.164190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.168321 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.166522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.168321 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.166988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.168321 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.167475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-config-out\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.170164 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.168719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-web-config\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.170164 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.169602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.170164 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.169682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.170724 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.170445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.170724 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.170532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.171538 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.171512    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.171627 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.171610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z652\" (UniqueName: \"kubernetes.io/projected/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-kube-api-access-2z652\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.172191 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.172167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.173922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.173878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.177177 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.176140    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.177177 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.176468    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.178324 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.178262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.179388 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.179368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-config\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.179572 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.179543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.264267 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.264208    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:31.368711 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.368256    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76ffffc8c9-5s826"]
Sep 16 21:23:31.368711 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.368358    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-db94579bd-tldhc"]
Sep 16 21:23:31.369062 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:23:31.369041    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1793a6c_4e04_461f_b8dd_d557ef233005.slice/crio-fbf5f11844c5b33d89b9e6ec3e0e60fad2fc5911ebabcf6e3907e84ea68b96a9 WatchSource:0}: Error finding container fbf5f11844c5b33d89b9e6ec3e0e60fad2fc5911ebabcf6e3907e84ea68b96a9: Status 404 returned error can't find the container with id fbf5f11844c5b33d89b9e6ec3e0e60fad2fc5911ebabcf6e3907e84ea68b96a9
Sep 16 21:23:31.370070 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:23:31.370047    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74def189_8a69_4b37_a34a_6302d0978328.slice/crio-bdda02172438aa6b62cea15f3fccdd6ea266d95a745d0705b2b785db2d97b99e WatchSource:0}: Error finding container bdda02172438aa6b62cea15f3fccdd6ea266d95a745d0705b2b785db2d97b99e: Status 404 returned error can't find the container with id bdda02172438aa6b62cea15f3fccdd6ea266d95a745d0705b2b785db2d97b99e
Sep 16 21:23:31.550246 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.550221    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 21:23:31.552160 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:23:31.552135    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54c2ddb_3cc6_46dd_8e3a_9e2cf01a4833.slice/crio-cd083a90e73e171a3d73a9cd50916c4f0741f21781d3df1549adbf79c358e5b5 WatchSource:0}: Error finding container cd083a90e73e171a3d73a9cd50916c4f0741f21781d3df1549adbf79c358e5b5: Status 404 returned error can't find the container with id cd083a90e73e171a3d73a9cd50916c4f0741f21781d3df1549adbf79c358e5b5
Sep 16 21:23:31.869714 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.869625    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-765979c47f-j8hjs"
Sep 16 21:23:31.959411 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.959376    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833","Type":"ContainerStarted","Data":"cd083a90e73e171a3d73a9cd50916c4f0741f21781d3df1549adbf79c358e5b5"}
Sep 16 21:23:31.960620 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.960591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db94579bd-tldhc" event={"ID":"a1793a6c-4e04-461f-b8dd-d557ef233005","Type":"ContainerStarted","Data":"fbf5f11844c5b33d89b9e6ec3e0e60fad2fc5911ebabcf6e3907e84ea68b96a9"}
Sep 16 21:23:31.962110 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.962064    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9pdvd" event={"ID":"b6ce6503-957e-48b0-836a-699b9dd71a86","Type":"ContainerStarted","Data":"ac94fe8f3594d2fb1d43c49cf5fbedd54e6f991f5730e4c16b5b8587e60a94d4"}
Sep 16 21:23:31.962233 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.962209    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:23:31.963712 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.963688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zxtz8" event={"ID":"8800530f-c408-44d0-a430-bd6d8cd8d95c","Type":"ContainerStarted","Data":"a5bce515a508c0a2e5b382889f1785b5c2c32c49dca470b31b5e9c45615182e5"}
Sep 16 21:23:31.965272 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.965251    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76ffffc8c9-5s826" event={"ID":"74def189-8a69-4b37-a34a-6302d0978328","Type":"ContainerStarted","Data":"fa0b2f87a4b87086f401784bdb18ad253b9b59f27c0402df02697456aa8a0435"}
Sep 16 21:23:31.965379 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.965287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76ffffc8c9-5s826" event={"ID":"74def189-8a69-4b37-a34a-6302d0978328","Type":"ContainerStarted","Data":"bdda02172438aa6b62cea15f3fccdd6ea266d95a745d0705b2b785db2d97b99e"}
Sep 16 21:23:31.977598 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.977556    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9pdvd" podStartSLOduration=66.610227627 podStartE2EDuration="1m11.977542328s" totalImagesPullingTime="5.367314701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:31.976901731 +0000 UTC m=+71.877111736" watchObservedRunningTime="2026-09-16 21:23:31.977542328 +0000 UTC m=+71.877752332"
Sep 16 21:23:31.992010 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:31.991947    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zxtz8" podStartSLOduration=66.619814234 podStartE2EDuration="1m11.991935499s" totalImagesPullingTime="5.372121265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:31.991474432 +0000 UTC m=+71.891684440" watchObservedRunningTime="2026-09-16 21:23:31.991935499 +0000 UTC m=+71.892145502"
Sep 16 21:23:32.016003 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:32.015950    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76ffffc8c9-5s826" podStartSLOduration=3.01593219 podStartE2EDuration="3.01593219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:32.015129889 +0000 UTC m=+71.915339895" watchObservedRunningTime="2026-09-16 21:23:32.01593219 +0000 UTC m=+71.916142193"
Sep 16 21:23:32.970898 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:32.970861    2563 generic.go:356] "Generic (PLEG): container finished" podID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerID="fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a" exitCode=0
Sep 16 21:23:32.971356 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:32.970909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833","Type":"ContainerDied","Data":"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a"}
Sep 16 21:23:33.978240 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:33.978197    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db94579bd-tldhc" event={"ID":"a1793a6c-4e04-461f-b8dd-d557ef233005","Type":"ContainerStarted","Data":"426ab6c1d65114adb6f1f96a492fe856fe26a73e71cc06137da7879c42d421c6"}
Sep 16 21:23:33.978733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:33.978306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db94579bd-tldhc" event={"ID":"a1793a6c-4e04-461f-b8dd-d557ef233005","Type":"ContainerStarted","Data":"2c7e55e9514ddcee1bbbf0155e96255777600457a45fdb10195785b305de5105"}
Sep 16 21:23:33.978733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:33.978336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db94579bd-tldhc" event={"ID":"a1793a6c-4e04-461f-b8dd-d557ef233005","Type":"ContainerStarted","Data":"38e32a388061354d5c8ca9281a2ce8488bb38276eda2543e030fd0022166f7d1"}
Sep 16 21:23:34.000311 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.000245    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-db94579bd-tldhc" podStartSLOduration=1.915599234 podStartE2EDuration="4.000219216s" totalImagesPullingTime="2.084619982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:34.0000059 +0000 UTC m=+73.900215936" watchObservedRunningTime="2026-09-16 21:23:34.000219216 +0000 UTC m=+73.900429236"
Sep 16 21:23:34.103761 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.103725    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76ffffc8c9-5s826"]
Sep 16 21:23:34.142570 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.142534    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-586c9cdcc9-mct8h"]
Sep 16 21:23:34.145709 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.145691    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.159196 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.159169    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586c9cdcc9-mct8h"]
Sep 16 21:23:34.287815 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.287720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-oauth-serving-cert\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.287815 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.287761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-serving-cert\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.287815 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.287791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-config\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.288102 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.287889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-service-ca\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.288102 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.287924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-trusted-ca-bundle\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.288102 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.287994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-oauth-config\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.288102 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.288023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4jfm\" (UniqueName: \"kubernetes.io/projected/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-kube-api-access-l4jfm\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.388786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.388748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-service-ca\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.388786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.388789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-trusted-ca-bundle\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.389024 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.388846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-oauth-config\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.389024 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.388877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4jfm\" (UniqueName: \"kubernetes.io/projected/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-kube-api-access-l4jfm\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.389024 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.388952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-oauth-serving-cert\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.389024 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.388982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-serving-cert\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.389381 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.389336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-config\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.389599 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.389574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-service-ca\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.390195 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.390136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-trusted-ca-bundle\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.390195 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.390162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-config\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.390340 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.390197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-oauth-serving-cert\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.392027 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.392007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-oauth-config\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.392323 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.392298    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-serving-cert\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.401631 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.401604    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4jfm\" (UniqueName: \"kubernetes.io/projected/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-kube-api-access-l4jfm\") pod \"console-586c9cdcc9-mct8h\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") " pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.454756 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.454712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:34.730519 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.730257    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586c9cdcc9-mct8h"]
Sep 16 21:23:34.733396 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:23:34.733356    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42ce972_d7b8_4352_bb70_28bbe9e6ab9f.slice/crio-9cea6c9bbf49fc9d2aa6e7532d31e5d78a42e67f563a3de79ddf591ed5d0ce96 WatchSource:0}: Error finding container 9cea6c9bbf49fc9d2aa6e7532d31e5d78a42e67f563a3de79ddf591ed5d0ce96: Status 404 returned error can't find the container with id 9cea6c9bbf49fc9d2aa6e7532d31e5d78a42e67f563a3de79ddf591ed5d0ce96
Sep 16 21:23:34.985776 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.985678    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586c9cdcc9-mct8h" event={"ID":"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f","Type":"ContainerStarted","Data":"d6fcabd0cfd99695271d1ccca19fed03a2c30c7b28c82694e6804b01123ea51b"}
Sep 16 21:23:34.986324 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:34.985784    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586c9cdcc9-mct8h" event={"ID":"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f","Type":"ContainerStarted","Data":"9cea6c9bbf49fc9d2aa6e7532d31e5d78a42e67f563a3de79ddf591ed5d0ce96"}
Sep 16 21:23:35.004578 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:35.004491    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-586c9cdcc9-mct8h" podStartSLOduration=1.004472698 podStartE2EDuration="1.004472698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:35.003937637 +0000 UTC m=+74.904147643" watchObservedRunningTime="2026-09-16 21:23:35.004472698 +0000 UTC m=+74.904682702"
Sep 16 21:23:36.997846 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:36.997810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833","Type":"ContainerStarted","Data":"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"}
Sep 16 21:23:36.998295 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:36.997890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833","Type":"ContainerStarted","Data":"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"}
Sep 16 21:23:39.008546 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:39.008504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833","Type":"ContainerStarted","Data":"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"}
Sep 16 21:23:39.008929 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:39.008585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833","Type":"ContainerStarted","Data":"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"}
Sep 16 21:23:39.008929 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:39.008628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833","Type":"ContainerStarted","Data":"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"}
Sep 16 21:23:39.008929 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:39.008682    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833","Type":"ContainerStarted","Data":"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"}
Sep 16 21:23:39.051336 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:39.051276    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9376510270000002 podStartE2EDuration="9.05125695s" totalImagesPullingTime="6.089331888s" totalInitContainerRuntime="24.274035ms" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:39.050278545 +0000 UTC m=+78.950488551" watchObservedRunningTime="2026-09-16 21:23:39.05125695 +0000 UTC m=+78.951466953"
Sep 16 21:23:40.112254 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:40.112204    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:41.264941 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:41.264886    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:23:44.455824 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:44.455687    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:44.455824 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:44.455756    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:44.460754 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:44.460729    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:45.035413 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:45.035381    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:23:45.090541 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:45.090504    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5756969dff-h294g"]
Sep 16 21:23:59.125768 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.125702    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76ffffc8c9-5s826" podUID="74def189-8a69-4b37-a34a-6302d0978328" containerName="console" containerID="cri-o://fa0b2f87a4b87086f401784bdb18ad253b9b59f27c0402df02697456aa8a0435" gracePeriod=15
Sep 16 21:23:59.373371 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.373340    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:23:59.510835 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.510794    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74def189-8a69-4b37-a34a-6302d0978328-console-serving-cert\") pod \"74def189-8a69-4b37-a34a-6302d0978328\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") "
Sep 16 21:23:59.510835 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.510833    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-console-config\") pod \"74def189-8a69-4b37-a34a-6302d0978328\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") "
Sep 16 21:23:59.511097 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.510854    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74def189-8a69-4b37-a34a-6302d0978328-console-oauth-config\") pod \"74def189-8a69-4b37-a34a-6302d0978328\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") "
Sep 16 21:23:59.511097 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.510876    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-oauth-serving-cert\") pod \"74def189-8a69-4b37-a34a-6302d0978328\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") "
Sep 16 21:23:59.511097 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.510895    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-service-ca\") pod \"74def189-8a69-4b37-a34a-6302d0978328\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") "
Sep 16 21:23:59.511097 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.510950    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-trusted-ca-bundle\") pod \"74def189-8a69-4b37-a34a-6302d0978328\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") "
Sep 16 21:23:59.511097 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.511016    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkclw\" (UniqueName: \"kubernetes.io/projected/74def189-8a69-4b37-a34a-6302d0978328-kube-api-access-zkclw\") pod \"74def189-8a69-4b37-a34a-6302d0978328\" (UID: \"74def189-8a69-4b37-a34a-6302d0978328\") "
Sep 16 21:23:59.511403 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.511367    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-console-config" (OuterVolumeSpecName: "console-config") pod "74def189-8a69-4b37-a34a-6302d0978328" (UID: "74def189-8a69-4b37-a34a-6302d0978328"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:23:59.511461 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.511379    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-service-ca" (OuterVolumeSpecName: "service-ca") pod "74def189-8a69-4b37-a34a-6302d0978328" (UID: "74def189-8a69-4b37-a34a-6302d0978328"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:23:59.511461 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.511382    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "74def189-8a69-4b37-a34a-6302d0978328" (UID: "74def189-8a69-4b37-a34a-6302d0978328"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:23:59.511570 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.511515    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "74def189-8a69-4b37-a34a-6302d0978328" (UID: "74def189-8a69-4b37-a34a-6302d0978328"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:23:59.513334 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.513308    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74def189-8a69-4b37-a34a-6302d0978328-kube-api-access-zkclw" (OuterVolumeSpecName: "kube-api-access-zkclw") pod "74def189-8a69-4b37-a34a-6302d0978328" (UID: "74def189-8a69-4b37-a34a-6302d0978328"). InnerVolumeSpecName "kube-api-access-zkclw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:23:59.513455 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.513329    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74def189-8a69-4b37-a34a-6302d0978328-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "74def189-8a69-4b37-a34a-6302d0978328" (UID: "74def189-8a69-4b37-a34a-6302d0978328"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:23:59.513455 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.513381    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74def189-8a69-4b37-a34a-6302d0978328-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "74def189-8a69-4b37-a34a-6302d0978328" (UID: "74def189-8a69-4b37-a34a-6302d0978328"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:23:59.611965 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.611921    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-trusted-ca-bundle\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:23:59.611965 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.611959    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkclw\" (UniqueName: \"kubernetes.io/projected/74def189-8a69-4b37-a34a-6302d0978328-kube-api-access-zkclw\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:23:59.611965 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.611973    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74def189-8a69-4b37-a34a-6302d0978328-console-serving-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:23:59.612210 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.611985    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-console-config\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:23:59.612210 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.611999    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74def189-8a69-4b37-a34a-6302d0978328-console-oauth-config\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:23:59.612210 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.612010    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-oauth-serving-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:23:59.612210 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:23:59.612022    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74def189-8a69-4b37-a34a-6302d0978328-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:00.092266 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:00.092232    2563 generic.go:356] "Generic (PLEG): container finished" podID="74def189-8a69-4b37-a34a-6302d0978328" containerID="fa0b2f87a4b87086f401784bdb18ad253b9b59f27c0402df02697456aa8a0435" exitCode=0
Sep 16 21:24:00.092434 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:00.092295    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76ffffc8c9-5s826"
Sep 16 21:24:00.092434 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:00.092314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76ffffc8c9-5s826" event={"ID":"74def189-8a69-4b37-a34a-6302d0978328","Type":"ContainerDied","Data":"fa0b2f87a4b87086f401784bdb18ad253b9b59f27c0402df02697456aa8a0435"}
Sep 16 21:24:00.092434 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:00.092388    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76ffffc8c9-5s826" event={"ID":"74def189-8a69-4b37-a34a-6302d0978328","Type":"ContainerDied","Data":"bdda02172438aa6b62cea15f3fccdd6ea266d95a745d0705b2b785db2d97b99e"}
Sep 16 21:24:00.092434 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:00.092409    2563 scope.go:117] "RemoveContainer" containerID="fa0b2f87a4b87086f401784bdb18ad253b9b59f27c0402df02697456aa8a0435"
Sep 16 21:24:00.101194 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:00.101177    2563 scope.go:117] "RemoveContainer" containerID="fa0b2f87a4b87086f401784bdb18ad253b9b59f27c0402df02697456aa8a0435"
Sep 16 21:24:00.101531 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:24:00.101509    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0b2f87a4b87086f401784bdb18ad253b9b59f27c0402df02697456aa8a0435\": container with ID starting with fa0b2f87a4b87086f401784bdb18ad253b9b59f27c0402df02697456aa8a0435 not found: ID does not exist" containerID="fa0b2f87a4b87086f401784bdb18ad253b9b59f27c0402df02697456aa8a0435"
Sep 16 21:24:00.101573 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:00.101541    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0b2f87a4b87086f401784bdb18ad253b9b59f27c0402df02697456aa8a0435"} err="failed to get container status \"fa0b2f87a4b87086f401784bdb18ad253b9b59f27c0402df02697456aa8a0435\": rpc error: code = NotFound desc = could not find container \"fa0b2f87a4b87086f401784bdb18ad253b9b59f27c0402df02697456aa8a0435\": container with ID starting with fa0b2f87a4b87086f401784bdb18ad253b9b59f27c0402df02697456aa8a0435 not found: ID does not exist"
Sep 16 21:24:00.112536 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:00.112511    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76ffffc8c9-5s826"]
Sep 16 21:24:00.116797 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:00.116775    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76ffffc8c9-5s826"]
Sep 16 21:24:00.606487 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:00.606453    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74def189-8a69-4b37-a34a-6302d0978328" path="/var/lib/kubelet/pods/74def189-8a69-4b37-a34a-6302d0978328/volumes"
Sep 16 21:24:02.973812 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:02.973781    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9pdvd"
Sep 16 21:24:10.110542 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.110501    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5756969dff-h294g" podUID="17fae843-8922-4f91-8c20-9e1abc4161b8" containerName="console" containerID="cri-o://d0fbd8ca966e5516fc105a9ffdd6c49f4708069eaebe0fa348bd3511891eb9f8" gracePeriod=15
Sep 16 21:24:10.350780 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.350755    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:24:10.508924 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.508886    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tqlt\" (UniqueName: \"kubernetes.io/projected/17fae843-8922-4f91-8c20-9e1abc4161b8-kube-api-access-9tqlt\") pod \"17fae843-8922-4f91-8c20-9e1abc4161b8\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") "
Sep 16 21:24:10.508924 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.508926    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-service-ca\") pod \"17fae843-8922-4f91-8c20-9e1abc4161b8\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") "
Sep 16 21:24:10.509185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.508947    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-trusted-ca-bundle\") pod \"17fae843-8922-4f91-8c20-9e1abc4161b8\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") "
Sep 16 21:24:10.509185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.508976    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17fae843-8922-4f91-8c20-9e1abc4161b8-console-oauth-config\") pod \"17fae843-8922-4f91-8c20-9e1abc4161b8\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") "
Sep 16 21:24:10.509185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.508993    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-console-config\") pod \"17fae843-8922-4f91-8c20-9e1abc4161b8\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") "
Sep 16 21:24:10.509185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.509012    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17fae843-8922-4f91-8c20-9e1abc4161b8-console-serving-cert\") pod \"17fae843-8922-4f91-8c20-9e1abc4161b8\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") "
Sep 16 21:24:10.509185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.509080    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-oauth-serving-cert\") pod \"17fae843-8922-4f91-8c20-9e1abc4161b8\" (UID: \"17fae843-8922-4f91-8c20-9e1abc4161b8\") "
Sep 16 21:24:10.509436 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.509380    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "17fae843-8922-4f91-8c20-9e1abc4161b8" (UID: "17fae843-8922-4f91-8c20-9e1abc4161b8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:24:10.509504 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.509465    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-console-config" (OuterVolumeSpecName: "console-config") pod "17fae843-8922-4f91-8c20-9e1abc4161b8" (UID: "17fae843-8922-4f91-8c20-9e1abc4161b8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:24:10.509594 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.509572    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "17fae843-8922-4f91-8c20-9e1abc4161b8" (UID: "17fae843-8922-4f91-8c20-9e1abc4161b8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:24:10.509682 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.509634    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-service-ca" (OuterVolumeSpecName: "service-ca") pod "17fae843-8922-4f91-8c20-9e1abc4161b8" (UID: "17fae843-8922-4f91-8c20-9e1abc4161b8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:24:10.511310 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.511284    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fae843-8922-4f91-8c20-9e1abc4161b8-kube-api-access-9tqlt" (OuterVolumeSpecName: "kube-api-access-9tqlt") pod "17fae843-8922-4f91-8c20-9e1abc4161b8" (UID: "17fae843-8922-4f91-8c20-9e1abc4161b8"). InnerVolumeSpecName "kube-api-access-9tqlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:24:10.511407 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.511290    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17fae843-8922-4f91-8c20-9e1abc4161b8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "17fae843-8922-4f91-8c20-9e1abc4161b8" (UID: "17fae843-8922-4f91-8c20-9e1abc4161b8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:10.511407 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.511366    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17fae843-8922-4f91-8c20-9e1abc4161b8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "17fae843-8922-4f91-8c20-9e1abc4161b8" (UID: "17fae843-8922-4f91-8c20-9e1abc4161b8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:10.610225 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.610196    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-oauth-serving-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:10.610225 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.610218    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tqlt\" (UniqueName: \"kubernetes.io/projected/17fae843-8922-4f91-8c20-9e1abc4161b8-kube-api-access-9tqlt\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:10.610225 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.610228    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:10.610454 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.610236    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-trusted-ca-bundle\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:10.610454 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.610248    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17fae843-8922-4f91-8c20-9e1abc4161b8-console-oauth-config\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:10.610454 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.610258    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17fae843-8922-4f91-8c20-9e1abc4161b8-console-config\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:10.610454 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:10.610266    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17fae843-8922-4f91-8c20-9e1abc4161b8-console-serving-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:11.135948 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:11.135912    2563 generic.go:356] "Generic (PLEG): container finished" podID="17fae843-8922-4f91-8c20-9e1abc4161b8" containerID="d0fbd8ca966e5516fc105a9ffdd6c49f4708069eaebe0fa348bd3511891eb9f8" exitCode=0
Sep 16 21:24:11.136335 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:11.135957    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5756969dff-h294g"
Sep 16 21:24:11.136335 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:11.136000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5756969dff-h294g" event={"ID":"17fae843-8922-4f91-8c20-9e1abc4161b8","Type":"ContainerDied","Data":"d0fbd8ca966e5516fc105a9ffdd6c49f4708069eaebe0fa348bd3511891eb9f8"}
Sep 16 21:24:11.136335 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:11.136046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5756969dff-h294g" event={"ID":"17fae843-8922-4f91-8c20-9e1abc4161b8","Type":"ContainerDied","Data":"26c557798de6a74eb19a62c46dde85051d3cf1935e9fc8a377dc6395f292e3ec"}
Sep 16 21:24:11.136335 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:11.136065    2563 scope.go:117] "RemoveContainer" containerID="d0fbd8ca966e5516fc105a9ffdd6c49f4708069eaebe0fa348bd3511891eb9f8"
Sep 16 21:24:11.151243 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:11.151216    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5756969dff-h294g"]
Sep 16 21:24:11.152856 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:11.152839    2563 scope.go:117] "RemoveContainer" containerID="d0fbd8ca966e5516fc105a9ffdd6c49f4708069eaebe0fa348bd3511891eb9f8"
Sep 16 21:24:11.153480 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:24:11.153387    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0fbd8ca966e5516fc105a9ffdd6c49f4708069eaebe0fa348bd3511891eb9f8\": container with ID starting with d0fbd8ca966e5516fc105a9ffdd6c49f4708069eaebe0fa348bd3511891eb9f8 not found: ID does not exist" containerID="d0fbd8ca966e5516fc105a9ffdd6c49f4708069eaebe0fa348bd3511891eb9f8"
Sep 16 21:24:11.153480 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:11.153444    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fbd8ca966e5516fc105a9ffdd6c49f4708069eaebe0fa348bd3511891eb9f8"} err="failed to get container status \"d0fbd8ca966e5516fc105a9ffdd6c49f4708069eaebe0fa348bd3511891eb9f8\": rpc error: code = NotFound desc = could not find container \"d0fbd8ca966e5516fc105a9ffdd6c49f4708069eaebe0fa348bd3511891eb9f8\": container with ID starting with d0fbd8ca966e5516fc105a9ffdd6c49f4708069eaebe0fa348bd3511891eb9f8 not found: ID does not exist"
Sep 16 21:24:11.154686 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:11.154668    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5756969dff-h294g"]
Sep 16 21:24:12.607318 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:12.607283    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17fae843-8922-4f91-8c20-9e1abc4161b8" path="/var/lib/kubelet/pods/17fae843-8922-4f91-8c20-9e1abc4161b8/volumes"
Sep 16 21:24:31.265148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:31.265107    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:31.283932 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:31.283904    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:32.239529 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:32.239503    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:43.792667 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.792618    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f5769c479-ztc62"]
Sep 16 21:24:43.793118 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.792961    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fae843-8922-4f91-8c20-9e1abc4161b8" containerName="console"
Sep 16 21:24:43.793118 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.792978    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fae843-8922-4f91-8c20-9e1abc4161b8" containerName="console"
Sep 16 21:24:43.793118 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.792994    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74def189-8a69-4b37-a34a-6302d0978328" containerName="console"
Sep 16 21:24:43.793118 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.793000    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="74def189-8a69-4b37-a34a-6302d0978328" containerName="console"
Sep 16 21:24:43.793118 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.793048    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="17fae843-8922-4f91-8c20-9e1abc4161b8" containerName="console"
Sep 16 21:24:43.793118 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.793057    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="74def189-8a69-4b37-a34a-6302d0978328" containerName="console"
Sep 16 21:24:43.795790 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.795774    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.807101 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.807075    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f5769c479-ztc62"]
Sep 16 21:24:43.860582 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.860551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-oauth-serving-cert\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.860755 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.860593    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a24cbed-58ee-410a-9c59-0da94389b32c-console-serving-cert\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.860755 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.860688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-service-ca\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.860755 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.860731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-trusted-ca-bundle\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.860857 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.860756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a24cbed-58ee-410a-9c59-0da94389b32c-console-oauth-config\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.860857 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.860778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csh66\" (UniqueName: \"kubernetes.io/projected/0a24cbed-58ee-410a-9c59-0da94389b32c-kube-api-access-csh66\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.860857 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.860820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-console-config\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.962103 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.962064    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-trusted-ca-bundle\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.962103 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.962104    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a24cbed-58ee-410a-9c59-0da94389b32c-console-oauth-config\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.962309 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.962123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csh66\" (UniqueName: \"kubernetes.io/projected/0a24cbed-58ee-410a-9c59-0da94389b32c-kube-api-access-csh66\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.962309 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.962164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-console-config\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.962309 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.962194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-oauth-serving-cert\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.962459 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.962384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a24cbed-58ee-410a-9c59-0da94389b32c-console-serving-cert\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.962513 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.962480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-service-ca\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.963079 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.963054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-console-config\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.963203 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.963100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-oauth-serving-cert\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.963203 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.963133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-trusted-ca-bundle\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.963203 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.963193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-service-ca\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.964759 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.964740    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a24cbed-58ee-410a-9c59-0da94389b32c-console-oauth-config\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.965002 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.964979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a24cbed-58ee-410a-9c59-0da94389b32c-console-serving-cert\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:43.969449 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:43.969430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csh66\" (UniqueName: \"kubernetes.io/projected/0a24cbed-58ee-410a-9c59-0da94389b32c-kube-api-access-csh66\") pod \"console-7f5769c479-ztc62\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") " pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:44.104935 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:44.104834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:44.377989 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:44.377908    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f5769c479-ztc62"]
Sep 16 21:24:44.380522 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:24:44.380493    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a24cbed_58ee_410a_9c59_0da94389b32c.slice/crio-8e0c2ea3b3c318a231ac03d569af3e6ca74fae4674aaad81bd303e8d5577909d WatchSource:0}: Error finding container 8e0c2ea3b3c318a231ac03d569af3e6ca74fae4674aaad81bd303e8d5577909d: Status 404 returned error can't find the container with id 8e0c2ea3b3c318a231ac03d569af3e6ca74fae4674aaad81bd303e8d5577909d
Sep 16 21:24:45.271034 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:45.270996    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5769c479-ztc62" event={"ID":"0a24cbed-58ee-410a-9c59-0da94389b32c","Type":"ContainerStarted","Data":"6c6528501fdc644375ee6404fb89d157e4b8f63baa173e6892dd5e83c4dbaf97"}
Sep 16 21:24:45.271436 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:45.271049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5769c479-ztc62" event={"ID":"0a24cbed-58ee-410a-9c59-0da94389b32c","Type":"ContainerStarted","Data":"8e0c2ea3b3c318a231ac03d569af3e6ca74fae4674aaad81bd303e8d5577909d"}
Sep 16 21:24:45.288754 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:45.288687    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f5769c479-ztc62" podStartSLOduration=2.288650271 podStartE2EDuration="2.288650271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:24:45.287340899 +0000 UTC m=+145.187550905" watchObservedRunningTime="2026-09-16 21:24:45.288650271 +0000 UTC m=+145.188860277"
Sep 16 21:24:49.292272 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.292239    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 21:24:49.292878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.292538    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="prometheus" containerID="cri-o://8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165" gracePeriod=600
Sep 16 21:24:49.292878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.292566    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="thanos-sidecar" containerID="cri-o://44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e" gracePeriod=600
Sep 16 21:24:49.292878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.292557    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="kube-rbac-proxy" containerID="cri-o://451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821" gracePeriod=600
Sep 16 21:24:49.292878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.292611    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3" gracePeriod=600
Sep 16 21:24:49.292878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.292678    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="config-reloader" containerID="cri-o://cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9" gracePeriod=600
Sep 16 21:24:49.292878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.292611    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="kube-rbac-proxy-web" containerID="cri-o://c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264" gracePeriod=600
Sep 16 21:24:49.526052 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.526027    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:49.610579 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610491    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-thanos-prometheus-http-client-file\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.610579 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610525    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-0\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.610579 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610545    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z652\" (UniqueName: \"kubernetes.io/projected/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-kube-api-access-2z652\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.610579 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610561    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-1\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.610918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610586    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-metrics-client-certs\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.610918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610630    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-config\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.610918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610704    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-metrics-client-ca\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.610918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610729    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-serving-certs-ca-bundle\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.610918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610758    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-trusted-ca-bundle\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.610918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610781    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.610918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610811    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-config-out\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.610918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610839    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.610918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610863    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-tls-assets\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.610918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610879    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-2\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.610918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610896    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-kubelet-serving-ca-bundle\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.610918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610918    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-db\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.611508 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610926    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:24:49.611508 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610945    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-web-config\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.611508 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.610971    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-tls\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.611508 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.611013    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-grpc-tls\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.611508 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.611046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-kube-rbac-proxy\") pod \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\" (UID: \"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833\") "
Sep 16 21:24:49.611508 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.611321    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.613436 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.611874    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:24:49.613436 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.612217    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:24:49.614388 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.613717    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:24:49.614388 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.614044    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:24:49.614388 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.614086    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:24:49.614599 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.614208    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:49.614599 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.614362    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:24:49.614599 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.614391    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-kube-api-access-2z652" (OuterVolumeSpecName: "kube-api-access-2z652") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "kube-api-access-2z652". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:24:49.614599 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.614408    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:24:49.614599 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.614523    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:49.614863 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.614813    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-config-out" (OuterVolumeSpecName: "config-out") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:24:49.615185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.615164    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:49.615633 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.615555    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:24:49.615633 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.615594    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:49.616594 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.616573    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-config" (OuterVolumeSpecName: "config") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:49.616878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.616860    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:49.617384 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.617366    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:49.617500 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.617477    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:49.626229 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.626207    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-web-config" (OuterVolumeSpecName: "web-config") pod "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" (UID: "f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:24:49.711711 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711675    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-config-out\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711711 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711704    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711711 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711716    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-tls-assets\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711726    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711735    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711744    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-db\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711752    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-web-config\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711761    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711769    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-grpc-tls\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711777    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-kube-rbac-proxy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711785    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711794    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711802    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z652\" (UniqueName: \"kubernetes.io/projected/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-kube-api-access-2z652\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711811    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-metrics-client-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711819    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-config\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711826    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-metrics-client-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711837    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711845    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:49.711927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:49.711854    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:24:50.301200 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.301163    2563 generic.go:356] "Generic (PLEG): container finished" podID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerID="a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3" exitCode=0
Sep 16 21:24:50.301200 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.301189    2563 generic.go:356] "Generic (PLEG): container finished" podID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerID="451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821" exitCode=0
Sep 16 21:24:50.301200 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.301198    2563 generic.go:356] "Generic (PLEG): container finished" podID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerID="c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264" exitCode=0
Sep 16 21:24:50.301200 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.301206    2563 generic.go:356] "Generic (PLEG): container finished" podID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerID="44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e" exitCode=0
Sep 16 21:24:50.301788 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.301212    2563 generic.go:356] "Generic (PLEG): container finished" podID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerID="cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9" exitCode=0
Sep 16 21:24:50.301788 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.301221    2563 generic.go:356] "Generic (PLEG): container finished" podID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerID="8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165" exitCode=0
Sep 16 21:24:50.301788 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.301240    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.301788 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.301241    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833","Type":"ContainerDied","Data":"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"}
Sep 16 21:24:50.301788 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.301377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833","Type":"ContainerDied","Data":"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"}
Sep 16 21:24:50.301788 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.301407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833","Type":"ContainerDied","Data":"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"}
Sep 16 21:24:50.301788 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.301439    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833","Type":"ContainerDied","Data":"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"}
Sep 16 21:24:50.301788 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.301451    2563 scope.go:117] "RemoveContainer" containerID="a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"
Sep 16 21:24:50.301788 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.301465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833","Type":"ContainerDied","Data":"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"}
Sep 16 21:24:50.301788 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.301489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833","Type":"ContainerDied","Data":"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"}
Sep 16 21:24:50.301788 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.301515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833","Type":"ContainerDied","Data":"cd083a90e73e171a3d73a9cd50916c4f0741f21781d3df1549adbf79c358e5b5"}
Sep 16 21:24:50.318505 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.318489    2563 scope.go:117] "RemoveContainer" containerID="451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"
Sep 16 21:24:50.325482 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.325457    2563 scope.go:117] "RemoveContainer" containerID="c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"
Sep 16 21:24:50.333923 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.333765    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 21:24:50.336413 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.336387    2563 scope.go:117] "RemoveContainer" containerID="44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"
Sep 16 21:24:50.338094 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.338077    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 21:24:50.343275 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.343253    2563 scope.go:117] "RemoveContainer" containerID="cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"
Sep 16 21:24:50.349938 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.349895    2563 scope.go:117] "RemoveContainer" containerID="8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"
Sep 16 21:24:50.357230 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.357198    2563 scope.go:117] "RemoveContainer" containerID="fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a"
Sep 16 21:24:50.358301 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358282    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 21:24:50.358556 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358543    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="thanos-sidecar"
Sep 16 21:24:50.358605 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358557    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="thanos-sidecar"
Sep 16 21:24:50.358605 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358568    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="kube-rbac-proxy"
Sep 16 21:24:50.358605 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358573    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="kube-rbac-proxy"
Sep 16 21:24:50.358605 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358581    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="config-reloader"
Sep 16 21:24:50.358605 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358586    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="config-reloader"
Sep 16 21:24:50.358605 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358592    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="kube-rbac-proxy-web"
Sep 16 21:24:50.358605 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358598    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="kube-rbac-proxy-web"
Sep 16 21:24:50.358605 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358606    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="prometheus"
Sep 16 21:24:50.358922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358612    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="prometheus"
Sep 16 21:24:50.358922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358623    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="init-config-reloader"
Sep 16 21:24:50.358922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358630    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="init-config-reloader"
Sep 16 21:24:50.358922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358635    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="kube-rbac-proxy-thanos"
Sep 16 21:24:50.358922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358640    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="kube-rbac-proxy-thanos"
Sep 16 21:24:50.358922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358701    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="kube-rbac-proxy"
Sep 16 21:24:50.358922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358709    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="kube-rbac-proxy-thanos"
Sep 16 21:24:50.358922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358714    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="thanos-sidecar"
Sep 16 21:24:50.358922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358719    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="kube-rbac-proxy-web"
Sep 16 21:24:50.358922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358724    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="prometheus"
Sep 16 21:24:50.358922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.358729    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" containerName="config-reloader"
Sep 16 21:24:50.363666 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.363571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.363729 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.363711    2563 scope.go:117] "RemoveContainer" containerID="a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"
Sep 16 21:24:50.364104 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:24:50.364067    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3\": container with ID starting with a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3 not found: ID does not exist" containerID="a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"
Sep 16 21:24:50.364201 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.364104    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"} err="failed to get container status \"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3\": rpc error: code = NotFound desc = could not find container \"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3\": container with ID starting with a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3 not found: ID does not exist"
Sep 16 21:24:50.364201 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.364124    2563 scope.go:117] "RemoveContainer" containerID="451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"
Sep 16 21:24:50.364362 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:24:50.364345    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821\": container with ID starting with 451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821 not found: ID does not exist" containerID="451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"
Sep 16 21:24:50.364409 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.364366    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"} err="failed to get container status \"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821\": rpc error: code = NotFound desc = could not find container \"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821\": container with ID starting with 451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821 not found: ID does not exist"
Sep 16 21:24:50.364409 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.364385    2563 scope.go:117] "RemoveContainer" containerID="c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"
Sep 16 21:24:50.364726 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:24:50.364694    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264\": container with ID starting with c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264 not found: ID does not exist" containerID="c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"
Sep 16 21:24:50.364726 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.364718    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"} err="failed to get container status \"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264\": rpc error: code = NotFound desc = could not find container \"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264\": container with ID starting with c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264 not found: ID does not exist"
Sep 16 21:24:50.364847 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.364732    2563 scope.go:117] "RemoveContainer" containerID="44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"
Sep 16 21:24:50.365018 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:24:50.365001    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e\": container with ID starting with 44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e not found: ID does not exist" containerID="44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"
Sep 16 21:24:50.365060 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.365024    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"} err="failed to get container status \"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e\": rpc error: code = NotFound desc = could not find container \"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e\": container with ID starting with 44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e not found: ID does not exist"
Sep 16 21:24:50.365060 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.365041    2563 scope.go:117] "RemoveContainer" containerID="cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"
Sep 16 21:24:50.365257 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:24:50.365238    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9\": container with ID starting with cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9 not found: ID does not exist" containerID="cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"
Sep 16 21:24:50.365333 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.365257    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"} err="failed to get container status \"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9\": rpc error: code = NotFound desc = could not find container \"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9\": container with ID starting with cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9 not found: ID does not exist"
Sep 16 21:24:50.365333 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.365270    2563 scope.go:117] "RemoveContainer" containerID="8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"
Sep 16 21:24:50.365511 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:24:50.365492    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165\": container with ID starting with 8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165 not found: ID does not exist" containerID="8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"
Sep 16 21:24:50.365577 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.365528    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"} err="failed to get container status \"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165\": rpc error: code = NotFound desc = could not find container \"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165\": container with ID starting with 8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165 not found: ID does not exist"
Sep 16 21:24:50.365577 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.365551    2563 scope.go:117] "RemoveContainer" containerID="fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a"
Sep 16 21:24:50.365896 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:24:50.365876    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a\": container with ID starting with fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a not found: ID does not exist" containerID="fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a"
Sep 16 21:24:50.365980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.365899    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a"} err="failed to get container status \"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a\": rpc error: code = NotFound desc = could not find container \"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a\": container with ID starting with fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a not found: ID does not exist"
Sep 16 21:24:50.365980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.365913    2563 scope.go:117] "RemoveContainer" containerID="a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"
Sep 16 21:24:50.365980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.365930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 21:24:50.366132 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.365983    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 21:24:50.366132 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.365989    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 21:24:50.366132 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366024    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 21:24:50.366286 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366205    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 21:24:50.366286 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366213    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 21:24:50.366286 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366211    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 21:24:50.366286 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366217    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 21:24:50.366286 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366247    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"} err="failed to get container status \"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3\": rpc error: code = NotFound desc = could not find container \"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3\": container with ID starting with a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3 not found: ID does not exist"
Sep 16 21:24:50.366286 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366269    2563 scope.go:117] "RemoveContainer" containerID="451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"
Sep 16 21:24:50.366566 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366370    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 21:24:50.366566 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366376    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 21:24:50.366566 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366445    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 21:24:50.366819 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366574    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"} err="failed to get container status \"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821\": rpc error: code = NotFound desc = could not find container \"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821\": container with ID starting with 451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821 not found: ID does not exist"
Sep 16 21:24:50.366819 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366617    2563 scope.go:117] "RemoveContainer" containerID="c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"
Sep 16 21:24:50.366819 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366629    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e80ii0cstljva\""
Sep 16 21:24:50.366979 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366894    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 21:24:50.366979 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366900    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rtjnz\""
Sep 16 21:24:50.366979 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366918    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"} err="failed to get container status \"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264\": rpc error: code = NotFound desc = could not find container \"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264\": container with ID starting with c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264 not found: ID does not exist"
Sep 16 21:24:50.366979 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366939    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 21:24:50.366979 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.366944    2563 scope.go:117] "RemoveContainer" containerID="44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"
Sep 16 21:24:50.367965 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.367937    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"} err="failed to get container status \"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e\": rpc error: code = NotFound desc = could not find container \"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e\": container with ID starting with 44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e not found: ID does not exist"
Sep 16 21:24:50.368090 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.367967    2563 scope.go:117] "RemoveContainer" containerID="cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"
Sep 16 21:24:50.368248 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.368228    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"} err="failed to get container status \"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9\": rpc error: code = NotFound desc = could not find container \"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9\": container with ID starting with cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9 not found: ID does not exist"
Sep 16 21:24:50.368320 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.368249    2563 scope.go:117] "RemoveContainer" containerID="8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"
Sep 16 21:24:50.368520 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.368496    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"} err="failed to get container status \"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165\": rpc error: code = NotFound desc = could not find container \"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165\": container with ID starting with 8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165 not found: ID does not exist"
Sep 16 21:24:50.368604 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.368522    2563 scope.go:117] "RemoveContainer" containerID="fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a"
Sep 16 21:24:50.368812 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.368784    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a"} err="failed to get container status \"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a\": rpc error: code = NotFound desc = could not find container \"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a\": container with ID starting with fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a not found: ID does not exist"
Sep 16 21:24:50.368900 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.368808    2563 scope.go:117] "RemoveContainer" containerID="a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"
Sep 16 21:24:50.369088 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.369064    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"} err="failed to get container status \"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3\": rpc error: code = NotFound desc = could not find container \"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3\": container with ID starting with a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3 not found: ID does not exist"
Sep 16 21:24:50.369137 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.369091    2563 scope.go:117] "RemoveContainer" containerID="451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"
Sep 16 21:24:50.369434 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.369414    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"} err="failed to get container status \"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821\": rpc error: code = NotFound desc = could not find container \"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821\": container with ID starting with 451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821 not found: ID does not exist"
Sep 16 21:24:50.369434 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.369433    2563 scope.go:117] "RemoveContainer" containerID="c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"
Sep 16 21:24:50.369568 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.369524    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 21:24:50.369703 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.369683    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"} err="failed to get container status \"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264\": rpc error: code = NotFound desc = could not find container \"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264\": container with ID starting with c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264 not found: ID does not exist"
Sep 16 21:24:50.369764 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.369705    2563 scope.go:117] "RemoveContainer" containerID="44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"
Sep 16 21:24:50.369975 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.369954    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"} err="failed to get container status \"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e\": rpc error: code = NotFound desc = could not find container \"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e\": container with ID starting with 44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e not found: ID does not exist"
Sep 16 21:24:50.369975 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.369973    2563 scope.go:117] "RemoveContainer" containerID="cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"
Sep 16 21:24:50.370276 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.370251    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"} err="failed to get container status \"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9\": rpc error: code = NotFound desc = could not find container \"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9\": container with ID starting with cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9 not found: ID does not exist"
Sep 16 21:24:50.370276 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.370275    2563 scope.go:117] "RemoveContainer" containerID="8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"
Sep 16 21:24:50.370593 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.370576    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"} err="failed to get container status \"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165\": rpc error: code = NotFound desc = could not find container \"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165\": container with ID starting with 8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165 not found: ID does not exist"
Sep 16 21:24:50.370593 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.370592    2563 scope.go:117] "RemoveContainer" containerID="fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a"
Sep 16 21:24:50.370942 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.370923    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a"} err="failed to get container status \"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a\": rpc error: code = NotFound desc = could not find container \"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a\": container with ID starting with fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a not found: ID does not exist"
Sep 16 21:24:50.371023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.370943    2563 scope.go:117] "RemoveContainer" containerID="a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"
Sep 16 21:24:50.371224 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.371190    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"} err="failed to get container status \"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3\": rpc error: code = NotFound desc = could not find container \"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3\": container with ID starting with a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3 not found: ID does not exist"
Sep 16 21:24:50.371224 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.371217    2563 scope.go:117] "RemoveContainer" containerID="451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"
Sep 16 21:24:50.371676 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.371523    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"} err="failed to get container status \"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821\": rpc error: code = NotFound desc = could not find container \"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821\": container with ID starting with 451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821 not found: ID does not exist"
Sep 16 21:24:50.371676 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.371549    2563 scope.go:117] "RemoveContainer" containerID="c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"
Sep 16 21:24:50.372008 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.371973    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"} err="failed to get container status \"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264\": rpc error: code = NotFound desc = could not find container \"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264\": container with ID starting with c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264 not found: ID does not exist"
Sep 16 21:24:50.372008 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.371990    2563 scope.go:117] "RemoveContainer" containerID="44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"
Sep 16 21:24:50.372252 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.372226    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"} err="failed to get container status \"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e\": rpc error: code = NotFound desc = could not find container \"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e\": container with ID starting with 44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e not found: ID does not exist"
Sep 16 21:24:50.372252 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.372248    2563 scope.go:117] "RemoveContainer" containerID="cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"
Sep 16 21:24:50.372379 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.372278    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 21:24:50.372509 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.372489    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"} err="failed to get container status \"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9\": rpc error: code = NotFound desc = could not find container \"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9\": container with ID starting with cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9 not found: ID does not exist"
Sep 16 21:24:50.372581 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.372510    2563 scope.go:117] "RemoveContainer" containerID="8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"
Sep 16 21:24:50.372795 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.372777    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"} err="failed to get container status \"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165\": rpc error: code = NotFound desc = could not find container \"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165\": container with ID starting with 8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165 not found: ID does not exist"
Sep 16 21:24:50.372849 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.372796    2563 scope.go:117] "RemoveContainer" containerID="fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a"
Sep 16 21:24:50.373034 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.373017    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a"} err="failed to get container status \"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a\": rpc error: code = NotFound desc = could not find container \"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a\": container with ID starting with fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a not found: ID does not exist"
Sep 16 21:24:50.373096 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.373036    2563 scope.go:117] "RemoveContainer" containerID="a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"
Sep 16 21:24:50.373302 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.373285    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"} err="failed to get container status \"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3\": rpc error: code = NotFound desc = could not find container \"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3\": container with ID starting with a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3 not found: ID does not exist"
Sep 16 21:24:50.373346 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.373303    2563 scope.go:117] "RemoveContainer" containerID="451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"
Sep 16 21:24:50.373518 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.373497    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"} err="failed to get container status \"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821\": rpc error: code = NotFound desc = could not find container \"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821\": container with ID starting with 451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821 not found: ID does not exist"
Sep 16 21:24:50.373586 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.373519    2563 scope.go:117] "RemoveContainer" containerID="c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"
Sep 16 21:24:50.373795 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.373775    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"} err="failed to get container status \"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264\": rpc error: code = NotFound desc = could not find container \"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264\": container with ID starting with c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264 not found: ID does not exist"
Sep 16 21:24:50.373858 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.373795    2563 scope.go:117] "RemoveContainer" containerID="44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"
Sep 16 21:24:50.374022 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.374004    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"} err="failed to get container status \"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e\": rpc error: code = NotFound desc = could not find container \"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e\": container with ID starting with 44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e not found: ID does not exist"
Sep 16 21:24:50.374061 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.374025    2563 scope.go:117] "RemoveContainer" containerID="cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"
Sep 16 21:24:50.374263 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.374242    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"} err="failed to get container status \"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9\": rpc error: code = NotFound desc = could not find container \"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9\": container with ID starting with cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9 not found: ID does not exist"
Sep 16 21:24:50.374263 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.374261    2563 scope.go:117] "RemoveContainer" containerID="8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"
Sep 16 21:24:50.374490 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.374475    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"} err="failed to get container status \"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165\": rpc error: code = NotFound desc = could not find container \"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165\": container with ID starting with 8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165 not found: ID does not exist"
Sep 16 21:24:50.374490 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.374489    2563 scope.go:117] "RemoveContainer" containerID="fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a"
Sep 16 21:24:50.374768 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.374749    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a"} err="failed to get container status \"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a\": rpc error: code = NotFound desc = could not find container \"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a\": container with ID starting with fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a not found: ID does not exist"
Sep 16 21:24:50.374768 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.374767    2563 scope.go:117] "RemoveContainer" containerID="a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"
Sep 16 21:24:50.375005 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.374988    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3"} err="failed to get container status \"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3\": rpc error: code = NotFound desc = could not find container \"a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3\": container with ID starting with a99a542ad964745c03547da35d7e163ba613efa4620489e46477eac366377fa3 not found: ID does not exist"
Sep 16 21:24:50.375005 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.375003    2563 scope.go:117] "RemoveContainer" containerID="451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"
Sep 16 21:24:50.375231 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.375209    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821"} err="failed to get container status \"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821\": rpc error: code = NotFound desc = could not find container \"451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821\": container with ID starting with 451c297fc41126d1c7c4a956f4332108be9d223ade7a219b9e9296d96f83c821 not found: ID does not exist"
Sep 16 21:24:50.375277 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.375231    2563 scope.go:117] "RemoveContainer" containerID="c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"
Sep 16 21:24:50.375434 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.375418    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264"} err="failed to get container status \"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264\": rpc error: code = NotFound desc = could not find container \"c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264\": container with ID starting with c39801d0121062bce1948cc71fc166b9e854f740fed0f07768249b046e733264 not found: ID does not exist"
Sep 16 21:24:50.375434 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.375433    2563 scope.go:117] "RemoveContainer" containerID="44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"
Sep 16 21:24:50.375642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.375625    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e"} err="failed to get container status \"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e\": rpc error: code = NotFound desc = could not find container \"44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e\": container with ID starting with 44c07d09f066a0fa1a616b749435140b051b2cb025f77bdeba7c23eea20dae0e not found: ID does not exist"
Sep 16 21:24:50.375707 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.375643    2563 scope.go:117] "RemoveContainer" containerID="cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"
Sep 16 21:24:50.375903 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.375886    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9"} err="failed to get container status \"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9\": rpc error: code = NotFound desc = could not find container \"cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9\": container with ID starting with cc7d614a1d3493466e0cdb789eb1b4f1105306c51188b28e88992ce8887f9cb9 not found: ID does not exist"
Sep 16 21:24:50.375903 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.375901    2563 scope.go:117] "RemoveContainer" containerID="8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"
Sep 16 21:24:50.376122 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.376105    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165"} err="failed to get container status \"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165\": rpc error: code = NotFound desc = could not find container \"8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165\": container with ID starting with 8138c912982305052cb1d9e82cce0a211bffd15298353f24b10bbed2344b9165 not found: ID does not exist"
Sep 16 21:24:50.376161 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.376123    2563 scope.go:117] "RemoveContainer" containerID="fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a"
Sep 16 21:24:50.376317 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.376302    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a"} err="failed to get container status \"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a\": rpc error: code = NotFound desc = could not find container \"fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a\": container with ID starting with fe950c50fcd26e59c164a9fbc5b7b26580074324fd2d5e32b9fee9b5f5e18a5a not found: ID does not exist"
Sep 16 21:24:50.383310 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.383291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 21:24:50.416499 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.416499 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1beb123-dc5e-435f-a028-81b28c797d82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.416704 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxbgz\" (UniqueName: \"kubernetes.io/projected/b1beb123-dc5e-435f-a028-81b28c797d82-kube-api-access-cxbgz\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.416704 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1beb123-dc5e-435f-a028-81b28c797d82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.416704 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-config\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.416704 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.416704 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.416704 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1beb123-dc5e-435f-a028-81b28c797d82-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.416902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.416902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.416902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.416902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416770    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.416902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.416902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.416902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.416902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.417155 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.417155 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.417155 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.417155 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.416978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.517371 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.517371 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.517621 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517391    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.517621 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.517621 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.517621 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517482    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.517621 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.517621 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.517621 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.517621 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.518031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.518031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1beb123-dc5e-435f-a028-81b28c797d82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.518031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxbgz\" (UniqueName: \"kubernetes.io/projected/b1beb123-dc5e-435f-a028-81b28c797d82-kube-api-access-cxbgz\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.518031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1beb123-dc5e-435f-a028-81b28c797d82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.518031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-config\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.518031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.518031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.518031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1beb123-dc5e-435f-a028-81b28c797d82-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.518031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.518031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.517941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.518543 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.518268    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1beb123-dc5e-435f-a028-81b28c797d82-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.518543 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.518499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.520752 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.520608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.520752 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.520712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.521179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.520781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.521179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.521046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-config\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.521315 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.521205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.521462 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.521439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.521845 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.521748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.521845 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.521769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.521845 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.521824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.522177 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.522149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.522335 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.522312    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.522439 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.522405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.522950 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.522929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1beb123-dc5e-435f-a028-81b28c797d82-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.523073 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.523051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.523140 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.523097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1beb123-dc5e-435f-a028-81b28c797d82-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.523915 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.523889    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1beb123-dc5e-435f-a028-81b28c797d82-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.524251 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.524231    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1beb123-dc5e-435f-a028-81b28c797d82-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.525562 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.525543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxbgz\" (UniqueName: \"kubernetes.io/projected/b1beb123-dc5e-435f-a028-81b28c797d82-kube-api-access-cxbgz\") pod \"prometheus-k8s-0\" (UID: \"b1beb123-dc5e-435f-a028-81b28c797d82\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.607153 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.607077    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833" path="/var/lib/kubelet/pods/f54c2ddb-3cc6-46dd-8e3a-9e2cf01a4833/volumes"
Sep 16 21:24:50.674380 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.674352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:24:50.953021 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:50.952947    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 21:24:50.955182 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:24:50.955151    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1beb123_dc5e_435f_a028_81b28c797d82.slice/crio-91b12abf9eafd21613d2f7c44a554bdb751721d18b426746ddd1a3fe91aaf653 WatchSource:0}: Error finding container 91b12abf9eafd21613d2f7c44a554bdb751721d18b426746ddd1a3fe91aaf653: Status 404 returned error can't find the container with id 91b12abf9eafd21613d2f7c44a554bdb751721d18b426746ddd1a3fe91aaf653
Sep 16 21:24:51.306423 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:51.306338    2563 generic.go:356] "Generic (PLEG): container finished" podID="b1beb123-dc5e-435f-a028-81b28c797d82" containerID="3c120f8cbd955961270235c03d963f0e196e1273a9af1abefbcc4f0ea6f549a8" exitCode=0
Sep 16 21:24:51.306823 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:51.306421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1beb123-dc5e-435f-a028-81b28c797d82","Type":"ContainerDied","Data":"3c120f8cbd955961270235c03d963f0e196e1273a9af1abefbcc4f0ea6f549a8"}
Sep 16 21:24:51.306823 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:51.306502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1beb123-dc5e-435f-a028-81b28c797d82","Type":"ContainerStarted","Data":"91b12abf9eafd21613d2f7c44a554bdb751721d18b426746ddd1a3fe91aaf653"}
Sep 16 21:24:52.314926 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:52.314889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1beb123-dc5e-435f-a028-81b28c797d82","Type":"ContainerStarted","Data":"c0f8e8bdba565adab6a2e9b9dda45fd567091a982f77c53a6b1eae1de385eee9"}
Sep 16 21:24:52.315286 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:52.315008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1beb123-dc5e-435f-a028-81b28c797d82","Type":"ContainerStarted","Data":"65be99e882989669ab0a32b1c09b5a8a94000ca6986808c37275b3422e03165a"}
Sep 16 21:24:52.315286 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:52.315049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1beb123-dc5e-435f-a028-81b28c797d82","Type":"ContainerStarted","Data":"e4d6e4dd16e92689688419dbf7d8cb26467297ae36ebb65f6caff89ec815cd62"}
Sep 16 21:24:52.315286 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:52.315085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1beb123-dc5e-435f-a028-81b28c797d82","Type":"ContainerStarted","Data":"950bcdf189b8350bd543aac3295428805d70ade8f8092cdaed32bb28f8925f62"}
Sep 16 21:24:52.315286 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:52.315120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1beb123-dc5e-435f-a028-81b28c797d82","Type":"ContainerStarted","Data":"15ac149ff09e6fec9233690d301aebf259b2eac3b09a065eebb07dcabe3f31f9"}
Sep 16 21:24:52.315286 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:52.315162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1beb123-dc5e-435f-a028-81b28c797d82","Type":"ContainerStarted","Data":"ee2abffe52e49fa11693752c6c8872fbdd629ec2daf2e5191c782f621a26208c"}
Sep 16 21:24:52.342819 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:52.342758    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.319878354 podStartE2EDuration="2.34273842s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.860066ms" isStatefulPod=false podCreationTimestamp="2026-09-16 21:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:24:52.341404553 +0000 UTC m=+152.241614558" watchObservedRunningTime="2026-09-16 21:24:52.34273842 +0000 UTC m=+152.242948423"
Sep 16 21:24:54.105386 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:54.105346    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:54.105798 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:54.105408    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:54.110152 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:54.110131    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:54.327000 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:54.326968    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:24:54.369430 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:54.369347    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586c9cdcc9-mct8h"]
Sep 16 21:24:55.674961 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:24:55.674925    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:25:19.388082 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.388036    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-586c9cdcc9-mct8h" podUID="b42ce972-d7b8-4352-bb70-28bbe9e6ab9f" containerName="console" containerID="cri-o://d6fcabd0cfd99695271d1ccca19fed03a2c30c7b28c82694e6804b01123ea51b" gracePeriod=15
Sep 16 21:25:19.626835 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.626808    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:25:19.672589 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.672555    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4jfm\" (UniqueName: \"kubernetes.io/projected/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-kube-api-access-l4jfm\") pod \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") "
Sep 16 21:25:19.672791 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.672596    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-oauth-serving-cert\") pod \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") "
Sep 16 21:25:19.672791 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.672637    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-oauth-config\") pod \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") "
Sep 16 21:25:19.672791 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.672750    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-service-ca\") pod \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") "
Sep 16 21:25:19.672791 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.672786    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-serving-cert\") pod \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") "
Sep 16 21:25:19.673023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.672824    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-config\") pod \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") "
Sep 16 21:25:19.673023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.672863    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-trusted-ca-bundle\") pod \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\" (UID: \"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f\") "
Sep 16 21:25:19.673267 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.673156    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b42ce972-d7b8-4352-bb70-28bbe9e6ab9f" (UID: "b42ce972-d7b8-4352-bb70-28bbe9e6ab9f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:25:19.673267 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.673207    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-service-ca" (OuterVolumeSpecName: "service-ca") pod "b42ce972-d7b8-4352-bb70-28bbe9e6ab9f" (UID: "b42ce972-d7b8-4352-bb70-28bbe9e6ab9f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:25:19.673415 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.673307    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-config" (OuterVolumeSpecName: "console-config") pod "b42ce972-d7b8-4352-bb70-28bbe9e6ab9f" (UID: "b42ce972-d7b8-4352-bb70-28bbe9e6ab9f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:25:19.673415 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.673333    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b42ce972-d7b8-4352-bb70-28bbe9e6ab9f" (UID: "b42ce972-d7b8-4352-bb70-28bbe9e6ab9f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:25:19.674809 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.674782    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-kube-api-access-l4jfm" (OuterVolumeSpecName: "kube-api-access-l4jfm") pod "b42ce972-d7b8-4352-bb70-28bbe9e6ab9f" (UID: "b42ce972-d7b8-4352-bb70-28bbe9e6ab9f"). InnerVolumeSpecName "kube-api-access-l4jfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:25:19.674946 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.674811    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b42ce972-d7b8-4352-bb70-28bbe9e6ab9f" (UID: "b42ce972-d7b8-4352-bb70-28bbe9e6ab9f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:25:19.675096 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.675075    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b42ce972-d7b8-4352-bb70-28bbe9e6ab9f" (UID: "b42ce972-d7b8-4352-bb70-28bbe9e6ab9f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:25:19.773863 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.773808    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-trusted-ca-bundle\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:25:19.773863 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.773855    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4jfm\" (UniqueName: \"kubernetes.io/projected/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-kube-api-access-l4jfm\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:25:19.773863 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.773867    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-oauth-serving-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:25:19.773863 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.773876    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-oauth-config\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:25:19.773863 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.773887    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:25:19.774172 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.773895    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-serving-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:25:19.774172 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:19.773904    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f-console-config\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:25:20.418116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:20.418081    2563 generic.go:356] "Generic (PLEG): container finished" podID="b42ce972-d7b8-4352-bb70-28bbe9e6ab9f" containerID="d6fcabd0cfd99695271d1ccca19fed03a2c30c7b28c82694e6804b01123ea51b" exitCode=0
Sep 16 21:25:20.418688 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:20.418127    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586c9cdcc9-mct8h"
Sep 16 21:25:20.418688 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:20.418174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586c9cdcc9-mct8h" event={"ID":"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f","Type":"ContainerDied","Data":"d6fcabd0cfd99695271d1ccca19fed03a2c30c7b28c82694e6804b01123ea51b"}
Sep 16 21:25:20.418688 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:20.418232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586c9cdcc9-mct8h" event={"ID":"b42ce972-d7b8-4352-bb70-28bbe9e6ab9f","Type":"ContainerDied","Data":"9cea6c9bbf49fc9d2aa6e7532d31e5d78a42e67f563a3de79ddf591ed5d0ce96"}
Sep 16 21:25:20.418688 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:20.418266    2563 scope.go:117] "RemoveContainer" containerID="d6fcabd0cfd99695271d1ccca19fed03a2c30c7b28c82694e6804b01123ea51b"
Sep 16 21:25:20.426564 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:20.426548    2563 scope.go:117] "RemoveContainer" containerID="d6fcabd0cfd99695271d1ccca19fed03a2c30c7b28c82694e6804b01123ea51b"
Sep 16 21:25:20.426827 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:25:20.426804    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6fcabd0cfd99695271d1ccca19fed03a2c30c7b28c82694e6804b01123ea51b\": container with ID starting with d6fcabd0cfd99695271d1ccca19fed03a2c30c7b28c82694e6804b01123ea51b not found: ID does not exist" containerID="d6fcabd0cfd99695271d1ccca19fed03a2c30c7b28c82694e6804b01123ea51b"
Sep 16 21:25:20.426898 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:20.426837    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fcabd0cfd99695271d1ccca19fed03a2c30c7b28c82694e6804b01123ea51b"} err="failed to get container status \"d6fcabd0cfd99695271d1ccca19fed03a2c30c7b28c82694e6804b01123ea51b\": rpc error: code = NotFound desc = could not find container \"d6fcabd0cfd99695271d1ccca19fed03a2c30c7b28c82694e6804b01123ea51b\": container with ID starting with d6fcabd0cfd99695271d1ccca19fed03a2c30c7b28c82694e6804b01123ea51b not found: ID does not exist"
Sep 16 21:25:20.435470 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:20.435448    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586c9cdcc9-mct8h"]
Sep 16 21:25:20.438078 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:20.438058    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-586c9cdcc9-mct8h"]
Sep 16 21:25:20.606936 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:20.606897    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b42ce972-d7b8-4352-bb70-28bbe9e6ab9f" path="/var/lib/kubelet/pods/b42ce972-d7b8-4352-bb70-28bbe9e6ab9f/volumes"
Sep 16 21:25:50.675321 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:50.675277    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:25:50.691077 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:50.691049    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:25:51.545035 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:25:51.545005    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 21:26:35.583955 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.583923    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2gxgt"]
Sep 16 21:26:35.584420 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.584199    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b42ce972-d7b8-4352-bb70-28bbe9e6ab9f" containerName="console"
Sep 16 21:26:35.584420 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.584211    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b42ce972-d7b8-4352-bb70-28bbe9e6ab9f" containerName="console"
Sep 16 21:26:35.584420 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.584253    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b42ce972-d7b8-4352-bb70-28bbe9e6ab9f" containerName="console"
Sep 16 21:26:35.586975 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.586959    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2gxgt"
Sep 16 21:26:35.589043 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.589003    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 21:26:35.589179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.589061    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 21:26:35.589737 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.589713    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7jwd7\""
Sep 16 21:26:35.594711 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.594685    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2gxgt"]
Sep 16 21:26:35.696855 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.696828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j565h\" (UniqueName: \"kubernetes.io/projected/01f3ee3b-ece7-46cf-a527-1ed3dea94beb-kube-api-access-j565h\") pod \"cert-manager-55b4d954d5-2gxgt\" (UID: \"01f3ee3b-ece7-46cf-a527-1ed3dea94beb\") " pod="cert-manager/cert-manager-55b4d954d5-2gxgt"
Sep 16 21:26:35.697006 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.696871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01f3ee3b-ece7-46cf-a527-1ed3dea94beb-bound-sa-token\") pod \"cert-manager-55b4d954d5-2gxgt\" (UID: \"01f3ee3b-ece7-46cf-a527-1ed3dea94beb\") " pod="cert-manager/cert-manager-55b4d954d5-2gxgt"
Sep 16 21:26:35.797957 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.797924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01f3ee3b-ece7-46cf-a527-1ed3dea94beb-bound-sa-token\") pod \"cert-manager-55b4d954d5-2gxgt\" (UID: \"01f3ee3b-ece7-46cf-a527-1ed3dea94beb\") " pod="cert-manager/cert-manager-55b4d954d5-2gxgt"
Sep 16 21:26:35.798091 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.798017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j565h\" (UniqueName: \"kubernetes.io/projected/01f3ee3b-ece7-46cf-a527-1ed3dea94beb-kube-api-access-j565h\") pod \"cert-manager-55b4d954d5-2gxgt\" (UID: \"01f3ee3b-ece7-46cf-a527-1ed3dea94beb\") " pod="cert-manager/cert-manager-55b4d954d5-2gxgt"
Sep 16 21:26:35.805730 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.805702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01f3ee3b-ece7-46cf-a527-1ed3dea94beb-bound-sa-token\") pod \"cert-manager-55b4d954d5-2gxgt\" (UID: \"01f3ee3b-ece7-46cf-a527-1ed3dea94beb\") " pod="cert-manager/cert-manager-55b4d954d5-2gxgt"
Sep 16 21:26:35.805856 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.805803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j565h\" (UniqueName: \"kubernetes.io/projected/01f3ee3b-ece7-46cf-a527-1ed3dea94beb-kube-api-access-j565h\") pod \"cert-manager-55b4d954d5-2gxgt\" (UID: \"01f3ee3b-ece7-46cf-a527-1ed3dea94beb\") " pod="cert-manager/cert-manager-55b4d954d5-2gxgt"
Sep 16 21:26:35.908966 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:35.908863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2gxgt"
Sep 16 21:26:36.178719 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:36.177944    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2gxgt"]
Sep 16 21:26:36.183962 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:26:36.183932    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01f3ee3b_ece7_46cf_a527_1ed3dea94beb.slice/crio-da247ac15ecf556156236acf6d71e8736b5eaaa5d70014a14df8bed725680daa WatchSource:0}: Error finding container da247ac15ecf556156236acf6d71e8736b5eaaa5d70014a14df8bed725680daa: Status 404 returned error can't find the container with id da247ac15ecf556156236acf6d71e8736b5eaaa5d70014a14df8bed725680daa
Sep 16 21:26:36.691259 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:36.691223    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2gxgt" event={"ID":"01f3ee3b-ece7-46cf-a527-1ed3dea94beb","Type":"ContainerStarted","Data":"da247ac15ecf556156236acf6d71e8736b5eaaa5d70014a14df8bed725680daa"}
Sep 16 21:26:39.706135 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:39.706097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2gxgt" event={"ID":"01f3ee3b-ece7-46cf-a527-1ed3dea94beb","Type":"ContainerStarted","Data":"029fdd0bf7eb8c6a4053553c9a87a23a6ae4b4c4ded96e86ba5f6aabf98f62ee"}
Sep 16 21:26:39.742185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:26:39.742081    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-2gxgt" podStartSLOduration=1.46896546 podStartE2EDuration="4.74206131s" totalImagesPullingTime="3.27309585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:26:39.741046862 +0000 UTC m=+259.641256866" watchObservedRunningTime="2026-09-16 21:26:39.74206131 +0000 UTC m=+259.642271313"
Sep 16 21:27:01.026172 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.026087    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"]
Sep 16 21:27:01.029567 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.029545    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.031701 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.031676    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 21:27:01.031951 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.031935    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 21:27:01.032458 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.032444    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 21:27:01.032540 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.032479    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 21:27:01.032610 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.032591    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lk5q9\""
Sep 16 21:27:01.048959 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.048925    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"]
Sep 16 21:27:01.117992 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.117945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-tvj4m\" (UID: \"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.118176 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.118038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0-tmp\") pod \"opendatahub-operator-controller-manager-df9d657c6-tvj4m\" (UID: \"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.118176 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.118089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0-manifests\") pod \"opendatahub-operator-controller-manager-df9d657c6-tvj4m\" (UID: \"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.118176 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.118135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-tvj4m\" (UID: \"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.118176 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.118152    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7n2v\" (UniqueName: \"kubernetes.io/projected/0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0-kube-api-access-b7n2v\") pod \"opendatahub-operator-controller-manager-df9d657c6-tvj4m\" (UID: \"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.219466 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.219426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-tvj4m\" (UID: \"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.219666 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.219478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0-tmp\") pod \"opendatahub-operator-controller-manager-df9d657c6-tvj4m\" (UID: \"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.219666 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.219499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0-manifests\") pod \"opendatahub-operator-controller-manager-df9d657c6-tvj4m\" (UID: \"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.219666 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.219550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-tvj4m\" (UID: \"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.219666 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.219566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7n2v\" (UniqueName: \"kubernetes.io/projected/0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0-kube-api-access-b7n2v\") pod \"opendatahub-operator-controller-manager-df9d657c6-tvj4m\" (UID: \"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.219977 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.219954    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0-manifests\") pod \"opendatahub-operator-controller-manager-df9d657c6-tvj4m\" (UID: \"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.221770 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.221743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0-tmp\") pod \"opendatahub-operator-controller-manager-df9d657c6-tvj4m\" (UID: \"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.222029 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.222010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-tvj4m\" (UID: \"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.222091 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.222070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-tvj4m\" (UID: \"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.229630 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.229604    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7n2v\" (UniqueName: \"kubernetes.io/projected/0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0-kube-api-access-b7n2v\") pod \"opendatahub-operator-controller-manager-df9d657c6-tvj4m\" (UID: \"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.339278 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.339176    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:01.622603 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.622424    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"]
Sep 16 21:27:01.625307 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:27:01.625277    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5765a4_8e84_49c6_8ce0_f9b20bedf8e0.slice/crio-03ab159e7dc3b495167cf561653d0206b037fabf678f1e13e4433c4afe8d95dd WatchSource:0}: Error finding container 03ab159e7dc3b495167cf561653d0206b037fabf678f1e13e4433c4afe8d95dd: Status 404 returned error can't find the container with id 03ab159e7dc3b495167cf561653d0206b037fabf678f1e13e4433c4afe8d95dd
Sep 16 21:27:01.784848 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:01.784813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m" event={"ID":"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0","Type":"ContainerStarted","Data":"03ab159e7dc3b495167cf561653d0206b037fabf678f1e13e4433c4afe8d95dd"}
Sep 16 21:27:04.798728 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:04.798690    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m" event={"ID":"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0","Type":"ContainerStarted","Data":"b141cea4505fb18820ad44a67a0bca0dde4c0cd514589d6519a94bdd51b430e5"}
Sep 16 21:27:05.804282 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:05.804248    2563 generic.go:356] "Generic (PLEG): container finished" podID="0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0" containerID="b141cea4505fb18820ad44a67a0bca0dde4c0cd514589d6519a94bdd51b430e5" exitCode=0
Sep 16 21:27:05.804643 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:05.804307    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m" event={"ID":"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0","Type":"ContainerDied","Data":"b141cea4505fb18820ad44a67a0bca0dde4c0cd514589d6519a94bdd51b430e5"}
Sep 16 21:27:06.811669 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:06.811555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m" event={"ID":"0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0","Type":"ContainerStarted","Data":"dcbecb9bdd04c24937199a64d7772852c657539f8abcc30a0f6cd8ef5ef462db"}
Sep 16 21:27:06.812122 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:06.811686    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:06.834235 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:06.834183    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m" podStartSLOduration=1.035070627 podStartE2EDuration="5.834162815s" totalImagesPullingTime="3.767503347s" totalInitContainerRuntime="1.031588841s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:27:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:27:06.833157114 +0000 UTC m=+286.733367129" watchObservedRunningTime="2026-09-16 21:27:06.834162815 +0000 UTC m=+286.734372818"
Sep 16 21:27:17.818086 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:17.818053    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-tvj4m"
Sep 16 21:27:20.383623 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.383588    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq"]
Sep 16 21:27:20.386770 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.386748    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq"
Sep 16 21:27:20.389134 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.389113    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 21:27:20.389604 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.389573    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 21:27:20.389895 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.389881    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 21:27:20.390109 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.390096    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 21:27:20.390647 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.390628    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8f4r8\""
Sep 16 21:27:20.402167 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.402140    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq"]
Sep 16 21:27:20.484439 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.484404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d679b692-a50b-4df9-9f32-42c34a9b58d5-tmp\") pod \"kube-auth-proxy-7f65bc968c-wszkq\" (UID: \"d679b692-a50b-4df9-9f32-42c34a9b58d5\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq"
Sep 16 21:27:20.484439 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.484440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zztf5\" (UniqueName: \"kubernetes.io/projected/d679b692-a50b-4df9-9f32-42c34a9b58d5-kube-api-access-zztf5\") pod \"kube-auth-proxy-7f65bc968c-wszkq\" (UID: \"d679b692-a50b-4df9-9f32-42c34a9b58d5\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq"
Sep 16 21:27:20.484637 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.484482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d679b692-a50b-4df9-9f32-42c34a9b58d5-tls-certs\") pod \"kube-auth-proxy-7f65bc968c-wszkq\" (UID: \"d679b692-a50b-4df9-9f32-42c34a9b58d5\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq"
Sep 16 21:27:20.528298 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.528266    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 21:27:20.585810 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.585647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d679b692-a50b-4df9-9f32-42c34a9b58d5-tmp\") pod \"kube-auth-proxy-7f65bc968c-wszkq\" (UID: \"d679b692-a50b-4df9-9f32-42c34a9b58d5\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq"
Sep 16 21:27:20.585810 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.585713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zztf5\" (UniqueName: \"kubernetes.io/projected/d679b692-a50b-4df9-9f32-42c34a9b58d5-kube-api-access-zztf5\") pod \"kube-auth-proxy-7f65bc968c-wszkq\" (UID: \"d679b692-a50b-4df9-9f32-42c34a9b58d5\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq"
Sep 16 21:27:20.585810 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.585775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d679b692-a50b-4df9-9f32-42c34a9b58d5-tls-certs\") pod \"kube-auth-proxy-7f65bc968c-wszkq\" (UID: \"d679b692-a50b-4df9-9f32-42c34a9b58d5\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq"
Sep 16 21:27:20.595099 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.587923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d679b692-a50b-4df9-9f32-42c34a9b58d5-tmp\") pod \"kube-auth-proxy-7f65bc968c-wszkq\" (UID: \"d679b692-a50b-4df9-9f32-42c34a9b58d5\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq"
Sep 16 21:27:20.595099 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.590894    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 21:27:20.595991 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:27:20.595972    2563 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 16 21:27:20.596041 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:27:20.596034    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d679b692-a50b-4df9-9f32-42c34a9b58d5-tls-certs podName:d679b692-a50b-4df9-9f32-42c34a9b58d5 nodeName:}" failed. No retries permitted until 2026-09-16 21:27:21.096018287 +0000 UTC m=+300.996228273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d679b692-a50b-4df9-9f32-42c34a9b58d5-tls-certs") pod "kube-auth-proxy-7f65bc968c-wszkq" (UID: "d679b692-a50b-4df9-9f32-42c34a9b58d5") : secret "kube-auth-proxy-tls" not found
Sep 16 21:27:20.600870 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.600848    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 21:27:20.611066 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.611045    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 21:27:20.622042 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:20.622015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zztf5\" (UniqueName: \"kubernetes.io/projected/d679b692-a50b-4df9-9f32-42c34a9b58d5-kube-api-access-zztf5\") pod \"kube-auth-proxy-7f65bc968c-wszkq\" (UID: \"d679b692-a50b-4df9-9f32-42c34a9b58d5\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq"
Sep 16 21:27:21.191037 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:21.190993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d679b692-a50b-4df9-9f32-42c34a9b58d5-tls-certs\") pod \"kube-auth-proxy-7f65bc968c-wszkq\" (UID: \"d679b692-a50b-4df9-9f32-42c34a9b58d5\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq"
Sep 16 21:27:21.193554 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:21.193517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d679b692-a50b-4df9-9f32-42c34a9b58d5-tls-certs\") pod \"kube-auth-proxy-7f65bc968c-wszkq\" (UID: \"d679b692-a50b-4df9-9f32-42c34a9b58d5\") " pod="openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq"
Sep 16 21:27:21.300595 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:21.300555    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8f4r8\""
Sep 16 21:27:21.307245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:21.307217    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq"
Sep 16 21:27:21.586856 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:21.586827    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq"]
Sep 16 21:27:21.589535 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:27:21.589494    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd679b692_a50b_4df9_9f32_42c34a9b58d5.slice/crio-67849ad044a45b020bab0454ec13c58ca2ece7f0a8c972b557083e2bc7157de2 WatchSource:0}: Error finding container 67849ad044a45b020bab0454ec13c58ca2ece7f0a8c972b557083e2bc7157de2: Status 404 returned error can't find the container with id 67849ad044a45b020bab0454ec13c58ca2ece7f0a8c972b557083e2bc7157de2
Sep 16 21:27:21.864477 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:21.864391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq" event={"ID":"d679b692-a50b-4df9-9f32-42c34a9b58d5","Type":"ContainerStarted","Data":"67849ad044a45b020bab0454ec13c58ca2ece7f0a8c972b557083e2bc7157de2"}
Sep 16 21:27:24.560281 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:24.560257    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 21:27:24.877411 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:24.877320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq" event={"ID":"d679b692-a50b-4df9-9f32-42c34a9b58d5","Type":"ContainerStarted","Data":"3bc6e428912ff53743e37155d561cea09e56253e2c0d12d11f0004822e3129a6"}
Sep 16 21:27:24.895749 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:24.895702    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f65bc968c-wszkq" podStartSLOduration=1.9303550029999998 podStartE2EDuration="4.895687061s" totalImagesPullingTime="2.965332058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:27:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:27:24.893923798 +0000 UTC m=+304.794133803" watchObservedRunningTime="2026-09-16 21:27:24.895687061 +0000 UTC m=+304.795897062"
Sep 16 21:27:26.910920 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:26.910880    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"]
Sep 16 21:27:26.914532 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:26.914511    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:26.916828 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:26.916803    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 21:27:26.916946 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:26.916849    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-trn2s\""
Sep 16 21:27:26.927998 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:26.927976    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"]
Sep 16 21:27:27.045581 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.045547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26de3a9-5d79-44e6-9fae-ebb703fb3020-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.045581 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.045582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26de3a9-5d79-44e6-9fae-ebb703fb3020-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.045838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.045600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26de3a9-5d79-44e6-9fae-ebb703fb3020-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.045838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.045721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wx4x\" (UniqueName: \"kubernetes.io/projected/f26de3a9-5d79-44e6-9fae-ebb703fb3020-kube-api-access-8wx4x\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.045838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.045759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26de3a9-5d79-44e6-9fae-ebb703fb3020-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.045838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.045774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26de3a9-5d79-44e6-9fae-ebb703fb3020-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.045838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.045822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26de3a9-5d79-44e6-9fae-ebb703fb3020-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.046013 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.045857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26de3a9-5d79-44e6-9fae-ebb703fb3020-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.046013 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.045883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26de3a9-5d79-44e6-9fae-ebb703fb3020-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.147226 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.147190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26de3a9-5d79-44e6-9fae-ebb703fb3020-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.147433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.147232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26de3a9-5d79-44e6-9fae-ebb703fb3020-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.147433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.147261    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26de3a9-5d79-44e6-9fae-ebb703fb3020-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.147433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.147320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wx4x\" (UniqueName: \"kubernetes.io/projected/f26de3a9-5d79-44e6-9fae-ebb703fb3020-kube-api-access-8wx4x\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.147433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.147376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26de3a9-5d79-44e6-9fae-ebb703fb3020-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.147433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.147399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26de3a9-5d79-44e6-9fae-ebb703fb3020-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.147433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.147425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26de3a9-5d79-44e6-9fae-ebb703fb3020-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.147838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.147452    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26de3a9-5d79-44e6-9fae-ebb703fb3020-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.147838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.147479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26de3a9-5d79-44e6-9fae-ebb703fb3020-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.147946 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.147925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26de3a9-5d79-44e6-9fae-ebb703fb3020-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.147998 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.147941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26de3a9-5d79-44e6-9fae-ebb703fb3020-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.148116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.148095    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26de3a9-5d79-44e6-9fae-ebb703fb3020-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.148221 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.148200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26de3a9-5d79-44e6-9fae-ebb703fb3020-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.148375 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.148361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26de3a9-5d79-44e6-9fae-ebb703fb3020-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.149824 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.149802    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26de3a9-5d79-44e6-9fae-ebb703fb3020-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.149982 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.149964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26de3a9-5d79-44e6-9fae-ebb703fb3020-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.158069 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.158046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26de3a9-5d79-44e6-9fae-ebb703fb3020-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.158240 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.158218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wx4x\" (UniqueName: \"kubernetes.io/projected/f26de3a9-5d79-44e6-9fae-ebb703fb3020-kube-api-access-8wx4x\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd\" (UID: \"f26de3a9-5d79-44e6-9fae-ebb703fb3020\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.224990 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.224955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:27.505589 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.501327    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"]
Sep 16 21:27:27.507283 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:27:27.507250    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26de3a9_5d79_44e6_9fae_ebb703fb3020.slice/crio-09e96dc5dcdf333d980263e103bdcc5a816c337e9d6826306857eae4d9a57fa9 WatchSource:0}: Error finding container 09e96dc5dcdf333d980263e103bdcc5a816c337e9d6826306857eae4d9a57fa9: Status 404 returned error can't find the container with id 09e96dc5dcdf333d980263e103bdcc5a816c337e9d6826306857eae4d9a57fa9
Sep 16 21:27:27.509299 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.509276    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 21:27:27.890827 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:27.890736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd" event={"ID":"f26de3a9-5d79-44e6-9fae-ebb703fb3020","Type":"ContainerStarted","Data":"09e96dc5dcdf333d980263e103bdcc5a816c337e9d6826306857eae4d9a57fa9"}
Sep 16 21:27:30.347148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:30.347104    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:27:30.347517 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:30.347197    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:27:30.347517 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:30.347245    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:27:30.905358 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:30.905304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd" event={"ID":"f26de3a9-5d79-44e6-9fae-ebb703fb3020","Type":"ContainerStarted","Data":"3b65ffbe000498b4f89f15667526926b9fc504ad816f873c158256cc47519c1b"}
Sep 16 21:27:30.926008 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:30.925945    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd" podStartSLOduration=2.0884954159999998 podStartE2EDuration="4.925922623s" totalImagesPullingTime="2.837427207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:27:30.925872924 +0000 UTC m=+310.826082930" watchObservedRunningTime="2026-09-16 21:27:30.925922623 +0000 UTC m=+310.826132624"
Sep 16 21:27:31.224995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.224960    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bjt9h"]
Sep 16 21:27:31.229593 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.229570    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:31.229751 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.229675    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-bjt9h"
Sep 16 21:27:31.230591 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.230569    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:31.231972 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.231953    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 16 21:27:31.232041 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.231991    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-fbmn9\""
Sep 16 21:27:31.237197 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.237172    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bjt9h"]
Sep 16 21:27:31.385395 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.385354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqv9h\" (UniqueName: \"kubernetes.io/projected/f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b-kube-api-access-zqv9h\") pod \"odh-model-controller-7998c67dc4-bjt9h\" (UID: \"f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b\") " pod="opendatahub/odh-model-controller-7998c67dc4-bjt9h"
Sep 16 21:27:31.385805 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.385441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b-cert\") pod \"odh-model-controller-7998c67dc4-bjt9h\" (UID: \"f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b\") " pod="opendatahub/odh-model-controller-7998c67dc4-bjt9h"
Sep 16 21:27:31.486328 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.486243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqv9h\" (UniqueName: \"kubernetes.io/projected/f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b-kube-api-access-zqv9h\") pod \"odh-model-controller-7998c67dc4-bjt9h\" (UID: \"f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b\") " pod="opendatahub/odh-model-controller-7998c67dc4-bjt9h"
Sep 16 21:27:31.486328 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.486320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b-cert\") pod \"odh-model-controller-7998c67dc4-bjt9h\" (UID: \"f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b\") " pod="opendatahub/odh-model-controller-7998c67dc4-bjt9h"
Sep 16 21:27:31.488963 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.488933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b-cert\") pod \"odh-model-controller-7998c67dc4-bjt9h\" (UID: \"f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b\") " pod="opendatahub/odh-model-controller-7998c67dc4-bjt9h"
Sep 16 21:27:31.494568 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.494539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqv9h\" (UniqueName: \"kubernetes.io/projected/f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b-kube-api-access-zqv9h\") pod \"odh-model-controller-7998c67dc4-bjt9h\" (UID: \"f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b\") " pod="opendatahub/odh-model-controller-7998c67dc4-bjt9h"
Sep 16 21:27:31.539843 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.539804    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-bjt9h"
Sep 16 21:27:31.812635 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.812548    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bjt9h"]
Sep 16 21:27:31.815837 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:27:31.815810    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69a6ecb_18e1_4bd7_82ba_d5baf04fd01b.slice/crio-7dd0d547cdf0c4807182f1276012d9684f5ad8c8a8d5720a1508a7e021ca7928 WatchSource:0}: Error finding container 7dd0d547cdf0c4807182f1276012d9684f5ad8c8a8d5720a1508a7e021ca7928: Status 404 returned error can't find the container with id 7dd0d547cdf0c4807182f1276012d9684f5ad8c8a8d5720a1508a7e021ca7928
Sep 16 21:27:31.913034 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.912989    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-bjt9h" event={"ID":"f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b","Type":"ContainerStarted","Data":"7dd0d547cdf0c4807182f1276012d9684f5ad8c8a8d5720a1508a7e021ca7928"}
Sep 16 21:27:31.913328 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.913301    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:31.914557 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.914537    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd"
Sep 16 21:27:31.935001 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.934973    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bvhxd"]
Sep 16 21:27:31.939618 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.939594    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bvhxd"
Sep 16 21:27:31.941996 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.941972    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 16 21:27:31.950360 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:31.950333    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bvhxd"]
Sep 16 21:27:32.093119 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:32.093023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6nvh\" (UniqueName: \"kubernetes.io/projected/7b8b352f-26b5-4ec0-a008-c1519d1ab4c1-kube-api-access-b6nvh\") pod \"model-serving-api-54d89bd5fc-bvhxd\" (UID: \"7b8b352f-26b5-4ec0-a008-c1519d1ab4c1\") " pod="opendatahub/model-serving-api-54d89bd5fc-bvhxd"
Sep 16 21:27:32.093119 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:32.093080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b8b352f-26b5-4ec0-a008-c1519d1ab4c1-tls-certs\") pod \"model-serving-api-54d89bd5fc-bvhxd\" (UID: \"7b8b352f-26b5-4ec0-a008-c1519d1ab4c1\") " pod="opendatahub/model-serving-api-54d89bd5fc-bvhxd"
Sep 16 21:27:32.194437 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:32.194400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6nvh\" (UniqueName: \"kubernetes.io/projected/7b8b352f-26b5-4ec0-a008-c1519d1ab4c1-kube-api-access-b6nvh\") pod \"model-serving-api-54d89bd5fc-bvhxd\" (UID: \"7b8b352f-26b5-4ec0-a008-c1519d1ab4c1\") " pod="opendatahub/model-serving-api-54d89bd5fc-bvhxd"
Sep 16 21:27:32.194642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:32.194448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b8b352f-26b5-4ec0-a008-c1519d1ab4c1-tls-certs\") pod \"model-serving-api-54d89bd5fc-bvhxd\" (UID: \"7b8b352f-26b5-4ec0-a008-c1519d1ab4c1\") " pod="opendatahub/model-serving-api-54d89bd5fc-bvhxd"
Sep 16 21:27:32.197154 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:32.197121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b8b352f-26b5-4ec0-a008-c1519d1ab4c1-tls-certs\") pod \"model-serving-api-54d89bd5fc-bvhxd\" (UID: \"7b8b352f-26b5-4ec0-a008-c1519d1ab4c1\") " pod="opendatahub/model-serving-api-54d89bd5fc-bvhxd"
Sep 16 21:27:32.202676 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:32.202630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6nvh\" (UniqueName: \"kubernetes.io/projected/7b8b352f-26b5-4ec0-a008-c1519d1ab4c1-kube-api-access-b6nvh\") pod \"model-serving-api-54d89bd5fc-bvhxd\" (UID: \"7b8b352f-26b5-4ec0-a008-c1519d1ab4c1\") " pod="opendatahub/model-serving-api-54d89bd5fc-bvhxd"
Sep 16 21:27:32.252801 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:32.252764    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bvhxd"
Sep 16 21:27:32.525846 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:32.525808    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bvhxd"]
Sep 16 21:27:32.528203 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:27:32.528169    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8b352f_26b5_4ec0_a008_c1519d1ab4c1.slice/crio-86d5bd24b975036574d54706073a7f01cfeb521a76390918bcc61076fbbcaeed WatchSource:0}: Error finding container 86d5bd24b975036574d54706073a7f01cfeb521a76390918bcc61076fbbcaeed: Status 404 returned error can't find the container with id 86d5bd24b975036574d54706073a7f01cfeb521a76390918bcc61076fbbcaeed
Sep 16 21:27:32.922861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:32.922813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bvhxd" event={"ID":"7b8b352f-26b5-4ec0-a008-c1519d1ab4c1","Type":"ContainerStarted","Data":"86d5bd24b975036574d54706073a7f01cfeb521a76390918bcc61076fbbcaeed"}
Sep 16 21:27:35.939269 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:35.939235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-bjt9h" event={"ID":"f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b","Type":"ContainerStarted","Data":"64b8ed07d51a00ab2423fd94024b72b2e50c6d4593fde1e215f5833ebad3e473"}
Sep 16 21:27:35.939747 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:35.939448    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-bjt9h"
Sep 16 21:27:35.940928 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:35.940899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bvhxd" event={"ID":"7b8b352f-26b5-4ec0-a008-c1519d1ab4c1","Type":"ContainerStarted","Data":"1672a677c7959dba7d1a2e2d5b2b4e113e5dc7883cc77ad728e76177ab3fd0a1"}
Sep 16 21:27:35.941060 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:35.941040    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-bvhxd"
Sep 16 21:27:35.954893 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:35.954835    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-bjt9h" podStartSLOduration=1.4939963729999999 podStartE2EDuration="4.954814127s" totalImagesPullingTime="3.460817754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:27:35.954748091 +0000 UTC m=+315.854958095" watchObservedRunningTime="2026-09-16 21:27:35.954814127 +0000 UTC m=+315.855024133"
Sep 16 21:27:35.969220 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:35.969170    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-bvhxd" podStartSLOduration=2.222931537 podStartE2EDuration="4.969156931s" totalImagesPullingTime="2.746225394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:27:35.968371972 +0000 UTC m=+315.868581977" watchObservedRunningTime="2026-09-16 21:27:35.969156931 +0000 UTC m=+315.869366951"
Sep 16 21:27:38.675891 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.675857    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"]
Sep 16 21:27:38.679146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.679129    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:27:38.682950 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.682927    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 21:27:38.683095 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.682935    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 21:27:38.683880 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.683855    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qlc8g\""
Sep 16 21:27:38.683880 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.683890    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 21:27:38.684051 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.683855    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 21:27:38.684051 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.683933    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 21:27:38.697350 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.697320    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"]
Sep 16 21:27:38.861976 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.861941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b8ad92e-86a7-4d1f-920f-b7c1117c920e-metrics-cert\") pod \"lws-controller-manager-5dfdd9df77-qht25\" (UID: \"9b8ad92e-86a7-4d1f-920f-b7c1117c920e\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:27:38.861976 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.861978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b8ad92e-86a7-4d1f-920f-b7c1117c920e-cert\") pod \"lws-controller-manager-5dfdd9df77-qht25\" (UID: \"9b8ad92e-86a7-4d1f-920f-b7c1117c920e\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:27:38.862176 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.862020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ll5d\" (UniqueName: \"kubernetes.io/projected/9b8ad92e-86a7-4d1f-920f-b7c1117c920e-kube-api-access-4ll5d\") pod \"lws-controller-manager-5dfdd9df77-qht25\" (UID: \"9b8ad92e-86a7-4d1f-920f-b7c1117c920e\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:27:38.862176 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.862041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b8ad92e-86a7-4d1f-920f-b7c1117c920e-manager-config\") pod \"lws-controller-manager-5dfdd9df77-qht25\" (UID: \"9b8ad92e-86a7-4d1f-920f-b7c1117c920e\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:27:38.963364 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.963280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b8ad92e-86a7-4d1f-920f-b7c1117c920e-manager-config\") pod \"lws-controller-manager-5dfdd9df77-qht25\" (UID: \"9b8ad92e-86a7-4d1f-920f-b7c1117c920e\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:27:38.963364 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.963352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b8ad92e-86a7-4d1f-920f-b7c1117c920e-metrics-cert\") pod \"lws-controller-manager-5dfdd9df77-qht25\" (UID: \"9b8ad92e-86a7-4d1f-920f-b7c1117c920e\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:27:38.963589 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.963371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b8ad92e-86a7-4d1f-920f-b7c1117c920e-cert\") pod \"lws-controller-manager-5dfdd9df77-qht25\" (UID: \"9b8ad92e-86a7-4d1f-920f-b7c1117c920e\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:27:38.963589 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.963411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ll5d\" (UniqueName: \"kubernetes.io/projected/9b8ad92e-86a7-4d1f-920f-b7c1117c920e-kube-api-access-4ll5d\") pod \"lws-controller-manager-5dfdd9df77-qht25\" (UID: \"9b8ad92e-86a7-4d1f-920f-b7c1117c920e\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:27:38.964177 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.964150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9b8ad92e-86a7-4d1f-920f-b7c1117c920e-manager-config\") pod \"lws-controller-manager-5dfdd9df77-qht25\" (UID: \"9b8ad92e-86a7-4d1f-920f-b7c1117c920e\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:27:38.966049 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.966024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b8ad92e-86a7-4d1f-920f-b7c1117c920e-cert\") pod \"lws-controller-manager-5dfdd9df77-qht25\" (UID: \"9b8ad92e-86a7-4d1f-920f-b7c1117c920e\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:27:38.966192 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.966170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b8ad92e-86a7-4d1f-920f-b7c1117c920e-metrics-cert\") pod \"lws-controller-manager-5dfdd9df77-qht25\" (UID: \"9b8ad92e-86a7-4d1f-920f-b7c1117c920e\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:27:38.974279 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.974254    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ll5d\" (UniqueName: \"kubernetes.io/projected/9b8ad92e-86a7-4d1f-920f-b7c1117c920e-kube-api-access-4ll5d\") pod \"lws-controller-manager-5dfdd9df77-qht25\" (UID: \"9b8ad92e-86a7-4d1f-920f-b7c1117c920e\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:27:38.988438 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:38.988414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:27:39.265550 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:39.265459    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"]
Sep 16 21:27:39.269267 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:27:39.269240    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8ad92e_86a7_4d1f_920f_b7c1117c920e.slice/crio-a400db8199c249d239e3c94e12575bb467dba93ca9128a1893827ca29f2c6026 WatchSource:0}: Error finding container a400db8199c249d239e3c94e12575bb467dba93ca9128a1893827ca29f2c6026: Status 404 returned error can't find the container with id a400db8199c249d239e3c94e12575bb467dba93ca9128a1893827ca29f2c6026
Sep 16 21:27:39.964767 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:39.964731    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25" event={"ID":"9b8ad92e-86a7-4d1f-920f-b7c1117c920e","Type":"ContainerStarted","Data":"a400db8199c249d239e3c94e12575bb467dba93ca9128a1893827ca29f2c6026"}
Sep 16 21:27:42.980398 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:42.980358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25" event={"ID":"9b8ad92e-86a7-4d1f-920f-b7c1117c920e","Type":"ContainerStarted","Data":"267f949b9ddd611b69849fd8db7b50218300d040252dad4dfeed0cfc3a771751"}
Sep 16 21:27:42.980885 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:42.980475    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:27:43.023527 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:43.023461    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25" podStartSLOduration=2.296949213 podStartE2EDuration="5.02344408s" totalImagesPullingTime="2.726494867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:27:43.022779716 +0000 UTC m=+322.922989721" watchObservedRunningTime="2026-09-16 21:27:43.02344408 +0000 UTC m=+322.923654084"
Sep 16 21:27:46.946557 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:46.946528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-bjt9h"
Sep 16 21:27:46.948669 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:46.948637    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-bvhxd"
Sep 16 21:27:53.987056 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:27:53.987026    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-qht25"
Sep 16 21:28:22.783899 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:22.783861    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f5769c479-ztc62"]
Sep 16 21:28:47.808172 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:47.808109    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f5769c479-ztc62" podUID="0a24cbed-58ee-410a-9c59-0da94389b32c" containerName="console" containerID="cri-o://6c6528501fdc644375ee6404fb89d157e4b8f63baa173e6892dd5e83c4dbaf97" gracePeriod=15
Sep 16 21:28:48.045124 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.045099    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:28:48.168001 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.167919    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-console-config\") pod \"0a24cbed-58ee-410a-9c59-0da94389b32c\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") "
Sep 16 21:28:48.168001 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.167983    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-oauth-serving-cert\") pod \"0a24cbed-58ee-410a-9c59-0da94389b32c\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") "
Sep 16 21:28:48.168001 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.168002    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csh66\" (UniqueName: \"kubernetes.io/projected/0a24cbed-58ee-410a-9c59-0da94389b32c-kube-api-access-csh66\") pod \"0a24cbed-58ee-410a-9c59-0da94389b32c\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") "
Sep 16 21:28:48.168277 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.168025    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-trusted-ca-bundle\") pod \"0a24cbed-58ee-410a-9c59-0da94389b32c\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") "
Sep 16 21:28:48.168277 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.168075    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a24cbed-58ee-410a-9c59-0da94389b32c-console-oauth-config\") pod \"0a24cbed-58ee-410a-9c59-0da94389b32c\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") "
Sep 16 21:28:48.168277 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.168094    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a24cbed-58ee-410a-9c59-0da94389b32c-console-serving-cert\") pod \"0a24cbed-58ee-410a-9c59-0da94389b32c\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") "
Sep 16 21:28:48.168277 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.168161    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-service-ca\") pod \"0a24cbed-58ee-410a-9c59-0da94389b32c\" (UID: \"0a24cbed-58ee-410a-9c59-0da94389b32c\") "
Sep 16 21:28:48.168473 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.168362    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-console-config" (OuterVolumeSpecName: "console-config") pod "0a24cbed-58ee-410a-9c59-0da94389b32c" (UID: "0a24cbed-58ee-410a-9c59-0da94389b32c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:28:48.168523 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.168467    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0a24cbed-58ee-410a-9c59-0da94389b32c" (UID: "0a24cbed-58ee-410a-9c59-0da94389b32c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:28:48.168523 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.168482    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0a24cbed-58ee-410a-9c59-0da94389b32c" (UID: "0a24cbed-58ee-410a-9c59-0da94389b32c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:28:48.168758 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.168735    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-service-ca" (OuterVolumeSpecName: "service-ca") pod "0a24cbed-58ee-410a-9c59-0da94389b32c" (UID: "0a24cbed-58ee-410a-9c59-0da94389b32c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:28:48.170384 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.170359    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a24cbed-58ee-410a-9c59-0da94389b32c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0a24cbed-58ee-410a-9c59-0da94389b32c" (UID: "0a24cbed-58ee-410a-9c59-0da94389b32c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:28:48.170486 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.170401    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a24cbed-58ee-410a-9c59-0da94389b32c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0a24cbed-58ee-410a-9c59-0da94389b32c" (UID: "0a24cbed-58ee-410a-9c59-0da94389b32c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:28:48.170486 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.170399    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a24cbed-58ee-410a-9c59-0da94389b32c-kube-api-access-csh66" (OuterVolumeSpecName: "kube-api-access-csh66") pod "0a24cbed-58ee-410a-9c59-0da94389b32c" (UID: "0a24cbed-58ee-410a-9c59-0da94389b32c"). InnerVolumeSpecName "kube-api-access-csh66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:28:48.265059 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.265025    2563 generic.go:356] "Generic (PLEG): container finished" podID="0a24cbed-58ee-410a-9c59-0da94389b32c" containerID="6c6528501fdc644375ee6404fb89d157e4b8f63baa173e6892dd5e83c4dbaf97" exitCode=0
Sep 16 21:28:48.265216 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.265072    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f5769c479-ztc62"
Sep 16 21:28:48.265216 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.265103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5769c479-ztc62" event={"ID":"0a24cbed-58ee-410a-9c59-0da94389b32c","Type":"ContainerDied","Data":"6c6528501fdc644375ee6404fb89d157e4b8f63baa173e6892dd5e83c4dbaf97"}
Sep 16 21:28:48.265216 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.265153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f5769c479-ztc62" event={"ID":"0a24cbed-58ee-410a-9c59-0da94389b32c","Type":"ContainerDied","Data":"8e0c2ea3b3c318a231ac03d569af3e6ca74fae4674aaad81bd303e8d5577909d"}
Sep 16 21:28:48.265216 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.265172    2563 scope.go:117] "RemoveContainer" containerID="6c6528501fdc644375ee6404fb89d157e4b8f63baa173e6892dd5e83c4dbaf97"
Sep 16 21:28:48.268878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.268856    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-console-config\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:28:48.268980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.268882    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-oauth-serving-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:28:48.268980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.268897    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csh66\" (UniqueName: \"kubernetes.io/projected/0a24cbed-58ee-410a-9c59-0da94389b32c-kube-api-access-csh66\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:28:48.268980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.268911    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-trusted-ca-bundle\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:28:48.268980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.268922    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a24cbed-58ee-410a-9c59-0da94389b32c-console-oauth-config\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:28:48.268980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.268930    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a24cbed-58ee-410a-9c59-0da94389b32c-console-serving-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:28:48.268980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.268938    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a24cbed-58ee-410a-9c59-0da94389b32c-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:28:48.273852 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.273836    2563 scope.go:117] "RemoveContainer" containerID="6c6528501fdc644375ee6404fb89d157e4b8f63baa173e6892dd5e83c4dbaf97"
Sep 16 21:28:48.274133 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:28:48.274112    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6528501fdc644375ee6404fb89d157e4b8f63baa173e6892dd5e83c4dbaf97\": container with ID starting with 6c6528501fdc644375ee6404fb89d157e4b8f63baa173e6892dd5e83c4dbaf97 not found: ID does not exist" containerID="6c6528501fdc644375ee6404fb89d157e4b8f63baa173e6892dd5e83c4dbaf97"
Sep 16 21:28:48.274189 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.274143    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6528501fdc644375ee6404fb89d157e4b8f63baa173e6892dd5e83c4dbaf97"} err="failed to get container status \"6c6528501fdc644375ee6404fb89d157e4b8f63baa173e6892dd5e83c4dbaf97\": rpc error: code = NotFound desc = could not find container \"6c6528501fdc644375ee6404fb89d157e4b8f63baa173e6892dd5e83c4dbaf97\": container with ID starting with 6c6528501fdc644375ee6404fb89d157e4b8f63baa173e6892dd5e83c4dbaf97 not found: ID does not exist"
Sep 16 21:28:48.288726 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.288701    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f5769c479-ztc62"]
Sep 16 21:28:48.295034 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.295011    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f5769c479-ztc62"]
Sep 16 21:28:48.611439 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:28:48.611407    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a24cbed-58ee-410a-9c59-0da94389b32c" path="/var/lib/kubelet/pods/0a24cbed-58ee-410a-9c59-0da94389b32c/volumes"
Sep 16 21:29:09.667420 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.667390    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4"]
Sep 16 21:29:09.667892 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.667721    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a24cbed-58ee-410a-9c59-0da94389b32c" containerName="console"
Sep 16 21:29:09.667892 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.667732    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a24cbed-58ee-410a-9c59-0da94389b32c" containerName="console"
Sep 16 21:29:09.667892 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.667792    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a24cbed-58ee-410a-9c59-0da94389b32c" containerName="console"
Sep 16 21:29:09.672020 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.671999    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4"
Sep 16 21:29:09.674904 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.674865    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 21:29:09.675024 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.674913    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 21:29:09.675024 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.674913    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 21:29:09.675713 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.675698    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hkhcz\""
Sep 16 21:29:09.675804 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.675728    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 21:29:09.679924 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.679904    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4"]
Sep 16 21:29:09.855086 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.855054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ed62904c-1291-4169-a246-3e1afc960125-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bbjf4\" (UID: \"ed62904c-1291-4169-a246-3e1afc960125\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4"
Sep 16 21:29:09.855266 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.855111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4bsm\" (UniqueName: \"kubernetes.io/projected/ed62904c-1291-4169-a246-3e1afc960125-kube-api-access-t4bsm\") pod \"kuadrant-console-plugin-fbdd4f445-bbjf4\" (UID: \"ed62904c-1291-4169-a246-3e1afc960125\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4"
Sep 16 21:29:09.855266 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.855145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed62904c-1291-4169-a246-3e1afc960125-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bbjf4\" (UID: \"ed62904c-1291-4169-a246-3e1afc960125\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4"
Sep 16 21:29:09.956273 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.956232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ed62904c-1291-4169-a246-3e1afc960125-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bbjf4\" (UID: \"ed62904c-1291-4169-a246-3e1afc960125\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4"
Sep 16 21:29:09.956438 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.956306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4bsm\" (UniqueName: \"kubernetes.io/projected/ed62904c-1291-4169-a246-3e1afc960125-kube-api-access-t4bsm\") pod \"kuadrant-console-plugin-fbdd4f445-bbjf4\" (UID: \"ed62904c-1291-4169-a246-3e1afc960125\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4"
Sep 16 21:29:09.956438 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.956350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed62904c-1291-4169-a246-3e1afc960125-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bbjf4\" (UID: \"ed62904c-1291-4169-a246-3e1afc960125\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4"
Sep 16 21:29:09.957039 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.957019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ed62904c-1291-4169-a246-3e1afc960125-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bbjf4\" (UID: \"ed62904c-1291-4169-a246-3e1afc960125\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4"
Sep 16 21:29:09.958885 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.958861    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed62904c-1291-4169-a246-3e1afc960125-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bbjf4\" (UID: \"ed62904c-1291-4169-a246-3e1afc960125\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4"
Sep 16 21:29:09.964492 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.964445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4bsm\" (UniqueName: \"kubernetes.io/projected/ed62904c-1291-4169-a246-3e1afc960125-kube-api-access-t4bsm\") pod \"kuadrant-console-plugin-fbdd4f445-bbjf4\" (UID: \"ed62904c-1291-4169-a246-3e1afc960125\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4"
Sep 16 21:29:09.982341 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:09.982318    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4"
Sep 16 21:29:10.250988 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:10.250917    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4"]
Sep 16 21:29:10.254145 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:29:10.254113    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded62904c_1291_4169_a246_3e1afc960125.slice/crio-19296bf44e6d2248e5168c32e2c8fc7d49830004cf11c2216bcbaa4c86c9fc89 WatchSource:0}: Error finding container 19296bf44e6d2248e5168c32e2c8fc7d49830004cf11c2216bcbaa4c86c9fc89: Status 404 returned error can't find the container with id 19296bf44e6d2248e5168c32e2c8fc7d49830004cf11c2216bcbaa4c86c9fc89
Sep 16 21:29:10.356429 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:10.356391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4" event={"ID":"ed62904c-1291-4169-a246-3e1afc960125","Type":"ContainerStarted","Data":"19296bf44e6d2248e5168c32e2c8fc7d49830004cf11c2216bcbaa4c86c9fc89"}
Sep 16 21:29:15.385417 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:15.385383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4" event={"ID":"ed62904c-1291-4169-a246-3e1afc960125","Type":"ContainerStarted","Data":"d8cbd39828844231402a0a3f782d5d49c738766c1e971d31e90dc41ed8dc45ca"}
Sep 16 21:29:15.402950 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:15.402900    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bbjf4" podStartSLOduration=2.022961517 podStartE2EDuration="6.402883215s" totalImagesPullingTime="4.379921698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:29:15.400984539 +0000 UTC m=+415.301194543" watchObservedRunningTime="2026-09-16 21:29:15.402883215 +0000 UTC m=+415.303093218"
Sep 16 21:29:48.546856 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.546780    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"]
Sep 16 21:29:48.569513 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.569489    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"]
Sep 16 21:29:48.569682 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.569594    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.571630 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.571608    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-r5g5g\""
Sep 16 21:29:48.655038 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.654998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac674de-6254-4602-8a24-e03a8a075fa9-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.655038 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.655039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac674de-6254-4602-8a24-e03a8a075fa9-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.655237 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.655078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac674de-6254-4602-8a24-e03a8a075fa9-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.655237 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.655182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac674de-6254-4602-8a24-e03a8a075fa9-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.655237 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.655203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac674de-6254-4602-8a24-e03a8a075fa9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.655237 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.655234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac674de-6254-4602-8a24-e03a8a075fa9-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.655365 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.655254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac674de-6254-4602-8a24-e03a8a075fa9-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.655365 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.655276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac674de-6254-4602-8a24-e03a8a075fa9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.655365 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.655294    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8f8x\" (UniqueName: \"kubernetes.io/projected/8ac674de-6254-4602-8a24-e03a8a075fa9-kube-api-access-n8f8x\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.755963 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.755925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac674de-6254-4602-8a24-e03a8a075fa9-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.755963 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.755961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac674de-6254-4602-8a24-e03a8a075fa9-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.756200 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.755977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac674de-6254-4602-8a24-e03a8a075fa9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.756200 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.755998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8f8x\" (UniqueName: \"kubernetes.io/projected/8ac674de-6254-4602-8a24-e03a8a075fa9-kube-api-access-n8f8x\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.756200 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.756032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac674de-6254-4602-8a24-e03a8a075fa9-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.756200 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.756058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac674de-6254-4602-8a24-e03a8a075fa9-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.756200 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.756085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac674de-6254-4602-8a24-e03a8a075fa9-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.756200 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.756171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac674de-6254-4602-8a24-e03a8a075fa9-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.756505 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.756305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac674de-6254-4602-8a24-e03a8a075fa9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.756505 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.756428    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac674de-6254-4602-8a24-e03a8a075fa9-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.756625 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.756509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac674de-6254-4602-8a24-e03a8a075fa9-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.756625 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.756610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac674de-6254-4602-8a24-e03a8a075fa9-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.756740 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.756725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac674de-6254-4602-8a24-e03a8a075fa9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.756781 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.756738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac674de-6254-4602-8a24-e03a8a075fa9-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.758398 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.758376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac674de-6254-4602-8a24-e03a8a075fa9-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.758751 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.758736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac674de-6254-4602-8a24-e03a8a075fa9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.763586 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.763563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac674de-6254-4602-8a24-e03a8a075fa9-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.763765 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.763747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8f8x\" (UniqueName: \"kubernetes.io/projected/8ac674de-6254-4602-8a24-e03a8a075fa9-kube-api-access-n8f8x\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bplw6\" (UID: \"8ac674de-6254-4602-8a24-e03a8a075fa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:48.880727 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:48.880625    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:49.152983 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:49.152947    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"]
Sep 16 21:29:49.155281 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:29:49.155253    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac674de_6254_4602_8a24_e03a8a075fa9.slice/crio-da6926d3b0a012b81f8cde97247a8f9343798b87f73a3a7fb88d931223d3b618 WatchSource:0}: Error finding container da6926d3b0a012b81f8cde97247a8f9343798b87f73a3a7fb88d931223d3b618: Status 404 returned error can't find the container with id da6926d3b0a012b81f8cde97247a8f9343798b87f73a3a7fb88d931223d3b618
Sep 16 21:29:49.157883 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:49.157850    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:29:49.157950 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:49.157918    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:29:49.157990 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:49.157950    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:29:49.529882 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:49.529845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6" event={"ID":"8ac674de-6254-4602-8a24-e03a8a075fa9","Type":"ContainerStarted","Data":"b2ec22a2e266009bf822043a10217dd7193b54f5105a2ac7c1ec3b4a64d720b4"}
Sep 16 21:29:49.530065 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:49.529899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6" event={"ID":"8ac674de-6254-4602-8a24-e03a8a075fa9","Type":"ContainerStarted","Data":"da6926d3b0a012b81f8cde97247a8f9343798b87f73a3a7fb88d931223d3b618"}
Sep 16 21:29:49.568571 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:49.568517    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6" podStartSLOduration=1.568501226 podStartE2EDuration="1.568501226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:29:49.565535843 +0000 UTC m=+449.465745851" watchObservedRunningTime="2026-09-16 21:29:49.568501226 +0000 UTC m=+449.468711229"
Sep 16 21:29:49.881349 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:49.881264    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:50.886431 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:50.886403    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:51.540154 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:51.540112    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:51.541208 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:51.541180    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:29:51.674283 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:51.674245    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-smhzs"]
Sep 16 21:29:51.677785 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:51.677763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-smhzs"
Sep 16 21:29:51.680336 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:51.680318    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6m8j5\""
Sep 16 21:29:51.685934 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:51.685910    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-smhzs"]
Sep 16 21:29:51.784010 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:51.783971    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc5b5\" (UniqueName: \"kubernetes.io/projected/b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9-kube-api-access-wc5b5\") pod \"authorino-7bcfd8f9cd-smhzs\" (UID: \"b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9\") " pod="kuadrant-system/authorino-7bcfd8f9cd-smhzs"
Sep 16 21:29:51.884521 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:51.884428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc5b5\" (UniqueName: \"kubernetes.io/projected/b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9-kube-api-access-wc5b5\") pod \"authorino-7bcfd8f9cd-smhzs\" (UID: \"b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9\") " pod="kuadrant-system/authorino-7bcfd8f9cd-smhzs"
Sep 16 21:29:51.897152 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:51.897125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc5b5\" (UniqueName: \"kubernetes.io/projected/b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9-kube-api-access-wc5b5\") pod \"authorino-7bcfd8f9cd-smhzs\" (UID: \"b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9\") " pod="kuadrant-system/authorino-7bcfd8f9cd-smhzs"
Sep 16 21:29:51.987360 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:51.987324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-smhzs"
Sep 16 21:29:52.258004 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:52.257972    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-smhzs"]
Sep 16 21:29:52.261092 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:29:52.261060    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb77bfe1c_81cf_4fe4_84b4_3ff5b41cf3d9.slice/crio-36a29a7ed891fe42fc9375ab9e3ed75da06660b4646ae392687e1085beb4de73 WatchSource:0}: Error finding container 36a29a7ed891fe42fc9375ab9e3ed75da06660b4646ae392687e1085beb4de73: Status 404 returned error can't find the container with id 36a29a7ed891fe42fc9375ab9e3ed75da06660b4646ae392687e1085beb4de73
Sep 16 21:29:52.546315 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:52.546228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-smhzs" event={"ID":"b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9","Type":"ContainerStarted","Data":"36a29a7ed891fe42fc9375ab9e3ed75da06660b4646ae392687e1085beb4de73"}
Sep 16 21:29:55.196869 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:55.196834    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-smhzs"]
Sep 16 21:29:55.565091 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:55.565054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-smhzs" event={"ID":"b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9","Type":"ContainerStarted","Data":"c137985d98d3e852b5f487d178ec795307b814ded322b06e766f39646b0976ab"}
Sep 16 21:29:55.578398 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:55.578343    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-smhzs" podStartSLOduration=1.737500368 podStartE2EDuration="4.578326832s" totalImagesPullingTime="2.840826464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:29:55.578224664 +0000 UTC m=+455.478434669" watchObservedRunningTime="2026-09-16 21:29:55.578326832 +0000 UTC m=+455.478536874"
Sep 16 21:29:56.569396 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:56.569340    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-smhzs" podUID="b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9" containerName="authorino" containerID="cri-o://c137985d98d3e852b5f487d178ec795307b814ded322b06e766f39646b0976ab" gracePeriod=30
Sep 16 21:29:56.813036 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:56.813014    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-smhzs"
Sep 16 21:29:56.935179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:56.935151    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc5b5\" (UniqueName: \"kubernetes.io/projected/b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9-kube-api-access-wc5b5\") pod \"b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9\" (UID: \"b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9\") "
Sep 16 21:29:56.937304 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:56.937281    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9-kube-api-access-wc5b5" (OuterVolumeSpecName: "kube-api-access-wc5b5") pod "b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9" (UID: "b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9"). InnerVolumeSpecName "kube-api-access-wc5b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:29:57.036461 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:57.036427    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc5b5\" (UniqueName: \"kubernetes.io/projected/b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9-kube-api-access-wc5b5\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:29:57.574437 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:57.574400    2563 generic.go:356] "Generic (PLEG): container finished" podID="b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9" containerID="c137985d98d3e852b5f487d178ec795307b814ded322b06e766f39646b0976ab" exitCode=0
Sep 16 21:29:57.574437 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:57.574441    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-smhzs"
Sep 16 21:29:57.574962 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:57.574470    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-smhzs" event={"ID":"b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9","Type":"ContainerDied","Data":"c137985d98d3e852b5f487d178ec795307b814ded322b06e766f39646b0976ab"}
Sep 16 21:29:57.574962 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:57.574500    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-smhzs" event={"ID":"b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9","Type":"ContainerDied","Data":"36a29a7ed891fe42fc9375ab9e3ed75da06660b4646ae392687e1085beb4de73"}
Sep 16 21:29:57.574962 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:57.574515    2563 scope.go:117] "RemoveContainer" containerID="c137985d98d3e852b5f487d178ec795307b814ded322b06e766f39646b0976ab"
Sep 16 21:29:57.583248 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:57.583233    2563 scope.go:117] "RemoveContainer" containerID="c137985d98d3e852b5f487d178ec795307b814ded322b06e766f39646b0976ab"
Sep 16 21:29:57.583505 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:29:57.583487    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c137985d98d3e852b5f487d178ec795307b814ded322b06e766f39646b0976ab\": container with ID starting with c137985d98d3e852b5f487d178ec795307b814ded322b06e766f39646b0976ab not found: ID does not exist" containerID="c137985d98d3e852b5f487d178ec795307b814ded322b06e766f39646b0976ab"
Sep 16 21:29:57.583542 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:57.583514    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c137985d98d3e852b5f487d178ec795307b814ded322b06e766f39646b0976ab"} err="failed to get container status \"c137985d98d3e852b5f487d178ec795307b814ded322b06e766f39646b0976ab\": rpc error: code = NotFound desc = could not find container \"c137985d98d3e852b5f487d178ec795307b814ded322b06e766f39646b0976ab\": container with ID starting with c137985d98d3e852b5f487d178ec795307b814ded322b06e766f39646b0976ab not found: ID does not exist"
Sep 16 21:29:57.593758 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:57.593736    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-smhzs"]
Sep 16 21:29:57.597717 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:57.597694    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-smhzs"]
Sep 16 21:29:58.607409 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:29:58.607373    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9" path="/var/lib/kubelet/pods/b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9/volumes"
Sep 16 21:30:20.819459 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:20.819382    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-rn5q9"]
Sep 16 21:30:20.819843 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:20.819733    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9" containerName="authorino"
Sep 16 21:30:20.819843 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:20.819748    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9" containerName="authorino"
Sep 16 21:30:20.819917 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:20.819848    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b77bfe1c-81cf-4fe4-84b4-3ff5b41cf3d9" containerName="authorino"
Sep 16 21:30:20.825018 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:20.824981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rn5q9"
Sep 16 21:30:20.828557 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:20.828524    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6m8j5\""
Sep 16 21:30:20.829043 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:20.829016    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-rn5q9"]
Sep 16 21:30:20.853639 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:20.853611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89qzq\" (UniqueName: \"kubernetes.io/projected/e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9-kube-api-access-89qzq\") pod \"authorino-557f876f69-rn5q9\" (UID: \"e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9\") " pod="kuadrant-system/authorino-557f876f69-rn5q9"
Sep 16 21:30:20.922769 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:20.922736    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-rn5q9"]
Sep 16 21:30:20.923025 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:30:20.923008    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-89qzq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-rn5q9" podUID="e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9"
Sep 16 21:30:20.954808 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:20.954774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89qzq\" (UniqueName: \"kubernetes.io/projected/e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9-kube-api-access-89qzq\") pod \"authorino-557f876f69-rn5q9\" (UID: \"e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9\") " pod="kuadrant-system/authorino-557f876f69-rn5q9"
Sep 16 21:30:20.964148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:20.964120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89qzq\" (UniqueName: \"kubernetes.io/projected/e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9-kube-api-access-89qzq\") pod \"authorino-557f876f69-rn5q9\" (UID: \"e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9\") " pod="kuadrant-system/authorino-557f876f69-rn5q9"
Sep 16 21:30:21.093899 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.093820    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79c845dff9-ptls7"]
Sep 16 21:30:21.097141 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.097123    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79c845dff9-ptls7"
Sep 16 21:30:21.099186 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.099169    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 21:30:21.103646 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.103621    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79c845dff9-ptls7"]
Sep 16 21:30:21.157425 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.157392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de4317a0-f59d-4251-a647-234a820a57d4-tls-cert\") pod \"authorino-79c845dff9-ptls7\" (UID: \"de4317a0-f59d-4251-a647-234a820a57d4\") " pod="kuadrant-system/authorino-79c845dff9-ptls7"
Sep 16 21:30:21.157425 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.157433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pphxl\" (UniqueName: \"kubernetes.io/projected/de4317a0-f59d-4251-a647-234a820a57d4-kube-api-access-pphxl\") pod \"authorino-79c845dff9-ptls7\" (UID: \"de4317a0-f59d-4251-a647-234a820a57d4\") " pod="kuadrant-system/authorino-79c845dff9-ptls7"
Sep 16 21:30:21.258738 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.258699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de4317a0-f59d-4251-a647-234a820a57d4-tls-cert\") pod \"authorino-79c845dff9-ptls7\" (UID: \"de4317a0-f59d-4251-a647-234a820a57d4\") " pod="kuadrant-system/authorino-79c845dff9-ptls7"
Sep 16 21:30:21.258738 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.258739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pphxl\" (UniqueName: \"kubernetes.io/projected/de4317a0-f59d-4251-a647-234a820a57d4-kube-api-access-pphxl\") pod \"authorino-79c845dff9-ptls7\" (UID: \"de4317a0-f59d-4251-a647-234a820a57d4\") " pod="kuadrant-system/authorino-79c845dff9-ptls7"
Sep 16 21:30:21.261177 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.261146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de4317a0-f59d-4251-a647-234a820a57d4-tls-cert\") pod \"authorino-79c845dff9-ptls7\" (UID: \"de4317a0-f59d-4251-a647-234a820a57d4\") " pod="kuadrant-system/authorino-79c845dff9-ptls7"
Sep 16 21:30:21.266167 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.266149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pphxl\" (UniqueName: \"kubernetes.io/projected/de4317a0-f59d-4251-a647-234a820a57d4-kube-api-access-pphxl\") pod \"authorino-79c845dff9-ptls7\" (UID: \"de4317a0-f59d-4251-a647-234a820a57d4\") " pod="kuadrant-system/authorino-79c845dff9-ptls7"
Sep 16 21:30:21.406949 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.406867    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79c845dff9-ptls7"
Sep 16 21:30:21.675129 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.675002    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79c845dff9-ptls7"]
Sep 16 21:30:21.677396 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:30:21.677359    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde4317a0_f59d_4251_a647_234a820a57d4.slice/crio-11f2b6adcbe7cd8d34d57f3ae565c69012bd7ea198be580fb73d60df4f57459b WatchSource:0}: Error finding container 11f2b6adcbe7cd8d34d57f3ae565c69012bd7ea198be580fb73d60df4f57459b: Status 404 returned error can't find the container with id 11f2b6adcbe7cd8d34d57f3ae565c69012bd7ea198be580fb73d60df4f57459b
Sep 16 21:30:21.680666 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.680632    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rn5q9"
Sep 16 21:30:21.689360 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.689339    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rn5q9"
Sep 16 21:30:21.764126 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.764096    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89qzq\" (UniqueName: \"kubernetes.io/projected/e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9-kube-api-access-89qzq\") pod \"e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9\" (UID: \"e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9\") "
Sep 16 21:30:21.766202 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.766176    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9-kube-api-access-89qzq" (OuterVolumeSpecName: "kube-api-access-89qzq") pod "e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9" (UID: "e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9"). InnerVolumeSpecName "kube-api-access-89qzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:30:21.864795 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:21.864758    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89qzq\" (UniqueName: \"kubernetes.io/projected/e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9-kube-api-access-89qzq\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:30:22.686160 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:22.686129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79c845dff9-ptls7" event={"ID":"de4317a0-f59d-4251-a647-234a820a57d4","Type":"ContainerStarted","Data":"11f2b6adcbe7cd8d34d57f3ae565c69012bd7ea198be580fb73d60df4f57459b"}
Sep 16 21:30:22.686160 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:22.686149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-rn5q9"
Sep 16 21:30:22.711405 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:22.711378    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-rn5q9"]
Sep 16 21:30:22.715182 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:22.715160    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-rn5q9"]
Sep 16 21:30:23.691883 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:23.691848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79c845dff9-ptls7" event={"ID":"de4317a0-f59d-4251-a647-234a820a57d4","Type":"ContainerStarted","Data":"f19d49c33910140262878f4671bf1362fbd9945242ffd3e6f21a355db116d8f5"}
Sep 16 21:30:23.707737 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:23.707689    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-79c845dff9-ptls7" podStartSLOduration=1.490674471 podStartE2EDuration="2.707673973s" totalImagesPullingTime="1.216999502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:30:23.705710718 +0000 UTC m=+483.605920721" watchObservedRunningTime="2026-09-16 21:30:23.707673973 +0000 UTC m=+483.607883978"
Sep 16 21:30:24.607418 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:30:24.607388    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9" path="/var/lib/kubelet/pods/e41e206c-cc2e-4a77-bda8-6ba9d0fa6bb9/volumes"
Sep 16 21:31:12.918479 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:12.918386    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ac674de-6254-4602-8a24-e03a8a075fa9" containerID="b2ec22a2e266009bf822043a10217dd7193b54f5105a2ac7c1ec3b4a64d720b4" exitCode=0
Sep 16 21:31:12.918479 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:12.918462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6" event={"ID":"8ac674de-6254-4602-8a24-e03a8a075fa9","Type":"ContainerDied","Data":"b2ec22a2e266009bf822043a10217dd7193b54f5105a2ac7c1ec3b4a64d720b4"}
Sep 16 21:31:12.919017 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:12.918835    2563 scope.go:117] "RemoveContainer" containerID="b2ec22a2e266009bf822043a10217dd7193b54f5105a2ac7c1ec3b4a64d720b4"
Sep 16 21:31:12.919748 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:12.919723    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:31:12.919806 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:12.919785    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:31:12.919840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:12.919820    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:31:13.925541 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:13.925456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6" event={"ID":"8ac674de-6254-4602-8a24-e03a8a075fa9","Type":"ContainerStarted","Data":"5df8cd5583a9b45cf71997d29d79630105e80146ed3b580ea18c4887f2f50b2d"}
Sep 16 21:31:14.881979 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:14.881930    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:31:14.882197 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:14.882096    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bplw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 16 21:31:14.882197 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:14.882168    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6" podUID="8ac674de-6254-4602-8a24-e03a8a075fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 16 21:31:15.881551 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:15.881510    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bplw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 16 21:31:15.882179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:15.882142    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6" podUID="8ac674de-6254-4602-8a24-e03a8a075fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 16 21:31:16.882062 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:16.882023    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bplw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 16 21:31:16.883041 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:16.882611    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6" podUID="8ac674de-6254-4602-8a24-e03a8a075fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 16 21:31:17.881411 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:17.881365    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bplw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 16 21:31:17.881735 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:17.881705    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6" podUID="8ac674de-6254-4602-8a24-e03a8a075fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 16 21:31:18.882674 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:18.882621    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bplw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 16 21:31:18.883201 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:18.882713    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6" podUID="8ac674de-6254-4602-8a24-e03a8a075fa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 16 21:31:19.885544 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:19.885512    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:31:19.886026 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:19.885866    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:31:19.886583 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:19.886562    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bplw6"
Sep 16 21:31:30.799241 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.799202    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"]
Sep 16 21:31:30.802620 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.802596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.806372 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.806342    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 21:31:30.806523 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.806408    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 21:31:30.806523 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.806342    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 16 21:31:30.806620 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.806536    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pjb85\""
Sep 16 21:31:30.886442 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.886402    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"]
Sep 16 21:31:30.889084 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.889052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d882622e-983f-4139-9090-ad5df1fd15a0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.889240 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.889092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d882622e-983f-4139-9090-ad5df1fd15a0-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.889240 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.889121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d882622e-983f-4139-9090-ad5df1fd15a0-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.889240 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.889144    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d882622e-983f-4139-9090-ad5df1fd15a0-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.889240 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.889173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d882622e-983f-4139-9090-ad5df1fd15a0-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.889240 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.889192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czrrw\" (UniqueName: \"kubernetes.io/projected/d882622e-983f-4139-9090-ad5df1fd15a0-kube-api-access-czrrw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.989898 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.989856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d882622e-983f-4139-9090-ad5df1fd15a0-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.989898 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.989906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d882622e-983f-4139-9090-ad5df1fd15a0-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.990148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.989936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d882622e-983f-4139-9090-ad5df1fd15a0-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.990148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.989957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czrrw\" (UniqueName: \"kubernetes.io/projected/d882622e-983f-4139-9090-ad5df1fd15a0-kube-api-access-czrrw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.990148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.990010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d882622e-983f-4139-9090-ad5df1fd15a0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.990148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.990046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d882622e-983f-4139-9090-ad5df1fd15a0-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.990354 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.990272    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d882622e-983f-4139-9090-ad5df1fd15a0-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.990453 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.990423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d882622e-983f-4139-9090-ad5df1fd15a0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.990529 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.990424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d882622e-983f-4139-9090-ad5df1fd15a0-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.992187 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.992169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d882622e-983f-4139-9090-ad5df1fd15a0-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:30.992593 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:30.992563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d882622e-983f-4139-9090-ad5df1fd15a0-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:31.001782 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:31.001756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czrrw\" (UniqueName: \"kubernetes.io/projected/d882622e-983f-4139-9090-ad5df1fd15a0-kube-api-access-czrrw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpg5n\" (UID: \"d882622e-983f-4139-9090-ad5df1fd15a0\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:31.112617 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:31.112522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:31.402205 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:31.402175    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"]
Sep 16 21:31:31.404603 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:31:31.404576    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd882622e_983f_4139_9090_ad5df1fd15a0.slice/crio-cf381b0906cee8de28f7045c045092a188c4b5691eaefe096e6e7e4e80d329e4 WatchSource:0}: Error finding container cf381b0906cee8de28f7045c045092a188c4b5691eaefe096e6e7e4e80d329e4: Status 404 returned error can't find the container with id cf381b0906cee8de28f7045c045092a188c4b5691eaefe096e6e7e4e80d329e4
Sep 16 21:31:32.007674 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:32.007322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n" event={"ID":"d882622e-983f-4139-9090-ad5df1fd15a0","Type":"ContainerStarted","Data":"cf381b0906cee8de28f7045c045092a188c4b5691eaefe096e6e7e4e80d329e4"}
Sep 16 21:31:33.013530 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:33.013492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n" event={"ID":"d882622e-983f-4139-9090-ad5df1fd15a0","Type":"ContainerStarted","Data":"d4f13ed8c728f3732d0ab73ef4b4cc4fd399943099a057b9cc6e1bd10e6f7300"}
Sep 16 21:31:33.037577 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:33.037520    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n" podStartSLOduration=1.9615121260000001 podStartE2EDuration="3.037501954s" totalImagesPullingTime="1.075989828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:31:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:31:33.036687625 +0000 UTC m=+552.936897628" watchObservedRunningTime="2026-09-16 21:31:33.037501954 +0000 UTC m=+552.937711960"
Sep 16 21:31:41.112992 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:41.112947    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:41.113550 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:41.113011    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:41.125890 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:41.125859    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:31:42.063856 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:31:42.063827    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpg5n"
Sep 16 21:32:24.678419 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.678388    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"]
Sep 16 21:32:24.681950 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.681933    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.684106 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.684082    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 16 21:32:24.690145 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.690121    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"]
Sep 16 21:32:24.771276 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.771239    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.771276 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.771271    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.771473 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.771297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.771473 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.771434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.771585 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.771473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.771640 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.771622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtrcl\" (UniqueName: \"kubernetes.io/projected/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-kube-api-access-gtrcl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.872569 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.872531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.872569 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.872568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.872818 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.872591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.872818 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.872672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.872818 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.872705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.872818 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.872737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtrcl\" (UniqueName: \"kubernetes.io/projected/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-kube-api-access-gtrcl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.873022 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.872951    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.873080 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.873020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.873080 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.873053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.875270 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.875188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.875467 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.875447    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.880572 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.880541    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtrcl\" (UniqueName: \"kubernetes.io/projected/5ca26678-e9fb-4fd7-a21e-7e49f0a3491f-kube-api-access-gtrcl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-t99c6\" (UID: \"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:24.991180 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:24.991097    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:25.264120 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:25.264032    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"]
Sep 16 21:32:25.266955 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:32:25.266931    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ca26678_e9fb_4fd7_a21e_7e49f0a3491f.slice/crio-184f71b17fce5e08aae17a916218dec6b989d1ed8c9f8005dfec2a7dea86b94e WatchSource:0}: Error finding container 184f71b17fce5e08aae17a916218dec6b989d1ed8c9f8005dfec2a7dea86b94e: Status 404 returned error can't find the container with id 184f71b17fce5e08aae17a916218dec6b989d1ed8c9f8005dfec2a7dea86b94e
Sep 16 21:32:26.250302 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:26.250270    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6" event={"ID":"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f","Type":"ContainerStarted","Data":"b8fc2c047686ef5a4f3c93d69b9cf166f810a6b111339e4c651035edf41258f9"}
Sep 16 21:32:26.250302 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:26.250307    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6" event={"ID":"5ca26678-e9fb-4fd7-a21e-7e49f0a3491f","Type":"ContainerStarted","Data":"184f71b17fce5e08aae17a916218dec6b989d1ed8c9f8005dfec2a7dea86b94e"}
Sep 16 21:32:26.266468 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:26.266421    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6" podStartSLOduration=2.082673159 podStartE2EDuration="2.266407211s" totalImagesPullingTime="183.734052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:32:26.26595389 +0000 UTC m=+606.166163896" watchObservedRunningTime="2026-09-16 21:32:26.266407211 +0000 UTC m=+606.166617215"
Sep 16 21:32:34.991454 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:34.991394    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:34.991969 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:34.991468    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:35.004310 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:35.004283    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:32:35.310280 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:32:35.310188    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-t99c6"
Sep 16 21:33:34.002210 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:34.002175    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b78788b8b-gp25s"]
Sep 16 21:33:34.005534 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:34.005512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b78788b8b-gp25s"
Sep 16 21:33:34.014028 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:34.014002    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b78788b8b-gp25s"]
Sep 16 21:33:34.084943 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:34.084907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljsws\" (UniqueName: \"kubernetes.io/projected/d4bfb8c5-4432-44ec-a798-5abf9401793b-kube-api-access-ljsws\") pod \"authorino-7b78788b8b-gp25s\" (UID: \"d4bfb8c5-4432-44ec-a798-5abf9401793b\") " pod="kuadrant-system/authorino-7b78788b8b-gp25s"
Sep 16 21:33:34.085102 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:34.084966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4bfb8c5-4432-44ec-a798-5abf9401793b-tls-cert\") pod \"authorino-7b78788b8b-gp25s\" (UID: \"d4bfb8c5-4432-44ec-a798-5abf9401793b\") " pod="kuadrant-system/authorino-7b78788b8b-gp25s"
Sep 16 21:33:34.186219 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:34.186189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljsws\" (UniqueName: \"kubernetes.io/projected/d4bfb8c5-4432-44ec-a798-5abf9401793b-kube-api-access-ljsws\") pod \"authorino-7b78788b8b-gp25s\" (UID: \"d4bfb8c5-4432-44ec-a798-5abf9401793b\") " pod="kuadrant-system/authorino-7b78788b8b-gp25s"
Sep 16 21:33:34.186389 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:34.186246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4bfb8c5-4432-44ec-a798-5abf9401793b-tls-cert\") pod \"authorino-7b78788b8b-gp25s\" (UID: \"d4bfb8c5-4432-44ec-a798-5abf9401793b\") " pod="kuadrant-system/authorino-7b78788b8b-gp25s"
Sep 16 21:33:34.188680 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:34.188621    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d4bfb8c5-4432-44ec-a798-5abf9401793b-tls-cert\") pod \"authorino-7b78788b8b-gp25s\" (UID: \"d4bfb8c5-4432-44ec-a798-5abf9401793b\") " pod="kuadrant-system/authorino-7b78788b8b-gp25s"
Sep 16 21:33:34.194756 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:34.194724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljsws\" (UniqueName: \"kubernetes.io/projected/d4bfb8c5-4432-44ec-a798-5abf9401793b-kube-api-access-ljsws\") pod \"authorino-7b78788b8b-gp25s\" (UID: \"d4bfb8c5-4432-44ec-a798-5abf9401793b\") " pod="kuadrant-system/authorino-7b78788b8b-gp25s"
Sep 16 21:33:34.314994 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:34.314912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b78788b8b-gp25s"
Sep 16 21:33:34.581134 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:34.581080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b78788b8b-gp25s"]
Sep 16 21:33:34.583524 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:33:34.583491    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4bfb8c5_4432_44ec_a798_5abf9401793b.slice/crio-59530ff10b4370117bf9c31679866d0c4a51970c834c7f4b4a5aa8909fcbc99f WatchSource:0}: Error finding container 59530ff10b4370117bf9c31679866d0c4a51970c834c7f4b4a5aa8909fcbc99f: Status 404 returned error can't find the container with id 59530ff10b4370117bf9c31679866d0c4a51970c834c7f4b4a5aa8909fcbc99f
Sep 16 21:33:34.585159 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:34.585142    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 21:33:35.564906 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:35.564871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b78788b8b-gp25s" event={"ID":"d4bfb8c5-4432-44ec-a798-5abf9401793b","Type":"ContainerStarted","Data":"59530ff10b4370117bf9c31679866d0c4a51970c834c7f4b4a5aa8909fcbc99f"}
Sep 16 21:33:36.571849 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:36.571809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b78788b8b-gp25s" event={"ID":"d4bfb8c5-4432-44ec-a798-5abf9401793b","Type":"ContainerStarted","Data":"70d629680f368c1090e69de66ccb9fd1f0448ad161a452516bd52e0f4e36a54b"}
Sep 16 21:33:36.586843 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:36.586788    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b78788b8b-gp25s" podStartSLOduration=2.466450174 podStartE2EDuration="3.586767193s" totalImagesPullingTime="1.120317019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:33:36.585131476 +0000 UTC m=+676.485341481" watchObservedRunningTime="2026-09-16 21:33:36.586767193 +0000 UTC m=+676.486977222"
Sep 16 21:33:36.621790 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:36.621753    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79c845dff9-ptls7"]
Sep 16 21:33:36.622010 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:36.621958    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-79c845dff9-ptls7" podUID="de4317a0-f59d-4251-a647-234a820a57d4" containerName="authorino" containerID="cri-o://f19d49c33910140262878f4671bf1362fbd9945242ffd3e6f21a355db116d8f5" gracePeriod=30
Sep 16 21:33:36.867476 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:36.867449    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79c845dff9-ptls7"
Sep 16 21:33:36.913932 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:36.913895    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de4317a0-f59d-4251-a647-234a820a57d4-tls-cert\") pod \"de4317a0-f59d-4251-a647-234a820a57d4\" (UID: \"de4317a0-f59d-4251-a647-234a820a57d4\") "
Sep 16 21:33:36.914085 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:36.913953    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pphxl\" (UniqueName: \"kubernetes.io/projected/de4317a0-f59d-4251-a647-234a820a57d4-kube-api-access-pphxl\") pod \"de4317a0-f59d-4251-a647-234a820a57d4\" (UID: \"de4317a0-f59d-4251-a647-234a820a57d4\") "
Sep 16 21:33:36.915994 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:36.915954    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4317a0-f59d-4251-a647-234a820a57d4-kube-api-access-pphxl" (OuterVolumeSpecName: "kube-api-access-pphxl") pod "de4317a0-f59d-4251-a647-234a820a57d4" (UID: "de4317a0-f59d-4251-a647-234a820a57d4"). InnerVolumeSpecName "kube-api-access-pphxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:33:36.923985 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:36.923957    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de4317a0-f59d-4251-a647-234a820a57d4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "de4317a0-f59d-4251-a647-234a820a57d4" (UID: "de4317a0-f59d-4251-a647-234a820a57d4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:33:37.015239 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:37.015199    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/de4317a0-f59d-4251-a647-234a820a57d4-tls-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:33:37.015239 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:37.015241    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pphxl\" (UniqueName: \"kubernetes.io/projected/de4317a0-f59d-4251-a647-234a820a57d4-kube-api-access-pphxl\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:33:37.578722 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:37.578685    2563 generic.go:356] "Generic (PLEG): container finished" podID="de4317a0-f59d-4251-a647-234a820a57d4" containerID="f19d49c33910140262878f4671bf1362fbd9945242ffd3e6f21a355db116d8f5" exitCode=0
Sep 16 21:33:37.579174 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:37.578732    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79c845dff9-ptls7"
Sep 16 21:33:37.579174 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:37.578767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79c845dff9-ptls7" event={"ID":"de4317a0-f59d-4251-a647-234a820a57d4","Type":"ContainerDied","Data":"f19d49c33910140262878f4671bf1362fbd9945242ffd3e6f21a355db116d8f5"}
Sep 16 21:33:37.579174 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:37.578817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79c845dff9-ptls7" event={"ID":"de4317a0-f59d-4251-a647-234a820a57d4","Type":"ContainerDied","Data":"11f2b6adcbe7cd8d34d57f3ae565c69012bd7ea198be580fb73d60df4f57459b"}
Sep 16 21:33:37.579174 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:37.578840    2563 scope.go:117] "RemoveContainer" containerID="f19d49c33910140262878f4671bf1362fbd9945242ffd3e6f21a355db116d8f5"
Sep 16 21:33:37.588002 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:37.587983    2563 scope.go:117] "RemoveContainer" containerID="f19d49c33910140262878f4671bf1362fbd9945242ffd3e6f21a355db116d8f5"
Sep 16 21:33:37.588268 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:33:37.588251    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19d49c33910140262878f4671bf1362fbd9945242ffd3e6f21a355db116d8f5\": container with ID starting with f19d49c33910140262878f4671bf1362fbd9945242ffd3e6f21a355db116d8f5 not found: ID does not exist" containerID="f19d49c33910140262878f4671bf1362fbd9945242ffd3e6f21a355db116d8f5"
Sep 16 21:33:37.588316 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:37.588275    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19d49c33910140262878f4671bf1362fbd9945242ffd3e6f21a355db116d8f5"} err="failed to get container status \"f19d49c33910140262878f4671bf1362fbd9945242ffd3e6f21a355db116d8f5\": rpc error: code = NotFound desc = could not find container \"f19d49c33910140262878f4671bf1362fbd9945242ffd3e6f21a355db116d8f5\": container with ID starting with f19d49c33910140262878f4671bf1362fbd9945242ffd3e6f21a355db116d8f5 not found: ID does not exist"
Sep 16 21:33:37.599699 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:37.599672    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79c845dff9-ptls7"]
Sep 16 21:33:37.602988 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:37.602969    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79c845dff9-ptls7"]
Sep 16 21:33:38.607716 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:38.607681    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de4317a0-f59d-4251-a647-234a820a57d4" path="/var/lib/kubelet/pods/de4317a0-f59d-4251-a647-234a820a57d4/volumes"
Sep 16 21:33:55.149838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.149806    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"]
Sep 16 21:33:55.150315 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.150127    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de4317a0-f59d-4251-a647-234a820a57d4" containerName="authorino"
Sep 16 21:33:55.150315 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.150138    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="de4317a0-f59d-4251-a647-234a820a57d4" containerName="authorino"
Sep 16 21:33:55.150315 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.150212    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="de4317a0-f59d-4251-a647-234a820a57d4" containerName="authorino"
Sep 16 21:33:55.153145 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.153129    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.155997 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.155972    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e5568d-openshift-default-dockercfg-c54qd\""
Sep 16 21:33:55.174014 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.173988    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"]
Sep 16 21:33:55.281406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.281368    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"]
Sep 16 21:33:55.285322 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.285296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.285913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.285889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-data\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.286003 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.285936    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-workload-socket\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.286003 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.285982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-workload-certs\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.286080 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.286054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-credential-socket\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.286114 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.286083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-podinfo\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.286148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.286115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-token\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.286148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.286134    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istiod-ca-cert\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.286209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.286159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-envoy\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.286209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.286186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s4vh\" (UniqueName: \"kubernetes.io/projected/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-kube-api-access-4s4vh\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.287641 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.287623    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e047d8-openshift-default-dockercfg-9nnn4\""
Sep 16 21:33:55.323534 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.323499    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"]
Sep 16 21:33:55.386782 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.386752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-workload-socket\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.386962 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.386793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-data\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.386962 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.386820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09e02b9-d226-419a-8a83-89efcd4825fc-istiod-ca-cert\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.386962 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.386921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-workload-certs\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.387107 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.386963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-envoy\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.387107 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.386989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thltf\" (UniqueName: \"kubernetes.io/projected/c09e02b9-d226-419a-8a83-89efcd4825fc-kube-api-access-thltf\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.387107 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-workload-certs\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.387270 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-credential-socket\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.387270 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-credential-socket\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.387270 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387184    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-workload-socket\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.387270 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-podinfo\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.387270 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-token\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.387270 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387239    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-workload-certs\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.387270 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istiod-ca-cert\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.387613 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-envoy\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.387613 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s4vh\" (UniqueName: \"kubernetes.io/projected/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-kube-api-access-4s4vh\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.387613 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-workload-socket\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.387613 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387428    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-credential-socket\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.387613 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-podinfo\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.387613 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-token\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.387613 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-data\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.387989 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.387902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-data\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.388084 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.388063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istiod-ca-cert\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.389996 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.389965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-envoy\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.390147 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.390130    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-podinfo\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.404910 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.404849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-token\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.405293 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.405276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s4vh\" (UniqueName: \"kubernetes.io/projected/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-kube-api-access-4s4vh\") pod \"e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") " pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.408449 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.408425    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"]
Sep 16 21:33:55.413057 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.413038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.415526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.415509    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-35f5fd-openshift-default-dockercfg-7dzc5\""
Sep 16 21:33:55.430926 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.430901    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"]
Sep 16 21:33:55.463040 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.463007    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:55.488478 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.488448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-podinfo\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.488618 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.488485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-token\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.488618 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.488516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vntds\" (UniqueName: \"kubernetes.io/projected/7f966d85-1c38-4502-bc7b-d36d78985f58-kube-api-access-vntds\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.488618 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.488548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-podinfo\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.488618 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.488564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-token\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.488838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.488722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-data\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.488838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.488768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09e02b9-d226-419a-8a83-89efcd4825fc-istiod-ca-cert\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.488838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.488795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-data\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.488838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.488825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-workload-certs\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.489084 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.488854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-envoy\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.489084 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.488883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thltf\" (UniqueName: \"kubernetes.io/projected/c09e02b9-d226-419a-8a83-89efcd4825fc-kube-api-access-thltf\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.489084 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.488907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-envoy\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.489084 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.488943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-workload-certs\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.489084 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.488986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-credential-socket\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.489084 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.489016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-workload-socket\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.489084 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.489070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f966d85-1c38-4502-bc7b-d36d78985f58-istiod-ca-cert\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.489444 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.489108    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-credential-socket\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.489444 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.489152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-workload-socket\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.489444 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.489216    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-workload-certs\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.489603 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.489487    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09e02b9-d226-419a-8a83-89efcd4825fc-istiod-ca-cert\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.489603 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.489494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-credential-socket\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.489603 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.489573    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-workload-socket\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.489775 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.489682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-data\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.491461 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.491424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-envoy\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.491643 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.491627    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-podinfo\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.497430 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.497408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-token\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.497697 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.497680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thltf\" (UniqueName: \"kubernetes.io/projected/c09e02b9-d226-419a-8a83-89efcd4825fc-kube-api-access-thltf\") pod \"e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") " pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.590713 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.590675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-data\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.590893 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.590726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-workload-certs\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.590893 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.590752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-envoy\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.590893 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.590792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-workload-socket\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.590893 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.590837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f966d85-1c38-4502-bc7b-d36d78985f58-istiod-ca-cert\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.590893 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.590872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-credential-socket\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.591173 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.590950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vntds\" (UniqueName: \"kubernetes.io/projected/7f966d85-1c38-4502-bc7b-d36d78985f58-kube-api-access-vntds\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.591173 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.591002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-podinfo\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.591173 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.591028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-token\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.591173 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.591074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-data\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.591340 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.591277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-workload-certs\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.591442 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.591420    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-credential-socket\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.591532 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.591514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-workload-socket\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.591720 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.591703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f966d85-1c38-4502-bc7b-d36d78985f58-istiod-ca-cert\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.593051 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.593034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-envoy\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.593584 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.593568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-podinfo\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.598061 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.598035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vntds\" (UniqueName: \"kubernetes.io/projected/7f966d85-1c38-4502-bc7b-d36d78985f58-kube-api-access-vntds\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.598205 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.598191    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:55.598580 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.598560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-token\") pod \"e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") " pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.727372 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.727339    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:55.737115 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.737087    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"]
Sep 16 21:33:55.738168 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:33:55.738126    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4b1a1b_eaa1_428c_b315_ca46e5deafa8.slice/crio-0af75074ac7710d4d3a18e8c50498939608676bb1ec093a195e2531e2689fec0 WatchSource:0}: Error finding container 0af75074ac7710d4d3a18e8c50498939608676bb1ec093a195e2531e2689fec0: Status 404 returned error can't find the container with id 0af75074ac7710d4d3a18e8c50498939608676bb1ec093a195e2531e2689fec0
Sep 16 21:33:55.740437 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.740406    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:33:55.740511 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.740486    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:33:55.740568 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.740534    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:33:55.874140 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.874104    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"]
Sep 16 21:33:55.877492 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:33:55.877455    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09e02b9_d226_419a_8a83_89efcd4825fc.slice/crio-e7c98fe4936224edd18a5fd7952e92120f7c4ab6d1c233eaef32e8860b8f4248 WatchSource:0}: Error finding container e7c98fe4936224edd18a5fd7952e92120f7c4ab6d1c233eaef32e8860b8f4248: Status 404 returned error can't find the container with id e7c98fe4936224edd18a5fd7952e92120f7c4ab6d1c233eaef32e8860b8f4248
Sep 16 21:33:55.880167 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.880130    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:33:55.880254 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.880202    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:33:55.880254 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:55.880232    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:33:56.009136 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:56.009049    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"]
Sep 16 21:33:56.014018 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:33:56.013987    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f966d85_1c38_4502_bc7b_d36d78985f58.slice/crio-762f3afb0534c3d52306f1387682696329a2cf6c9a7ba82f5f0aa20c237fce74 WatchSource:0}: Error finding container 762f3afb0534c3d52306f1387682696329a2cf6c9a7ba82f5f0aa20c237fce74: Status 404 returned error can't find the container with id 762f3afb0534c3d52306f1387682696329a2cf6c9a7ba82f5f0aa20c237fce74
Sep 16 21:33:56.016986 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:56.016937    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:33:56.017092 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:56.017055    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:33:56.017145 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:56.017105    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:33:56.670440 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:56.670399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr" event={"ID":"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8","Type":"ContainerStarted","Data":"c2095f774fd1e1360787e43484333f46f8b507dc8a6bf21dca4195453c28ae0d"}
Sep 16 21:33:56.670918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:56.670465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr" event={"ID":"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8","Type":"ContainerStarted","Data":"0af75074ac7710d4d3a18e8c50498939608676bb1ec093a195e2531e2689fec0"}
Sep 16 21:33:56.672057 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:56.672025    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m" event={"ID":"7f966d85-1c38-4502-bc7b-d36d78985f58","Type":"ContainerStarted","Data":"5dbad252c3a97baa344fc3cf5187ed474fa2dd7cd1ba717e3cb57e85ef212422"}
Sep 16 21:33:56.672178 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:56.672068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m" event={"ID":"7f966d85-1c38-4502-bc7b-d36d78985f58","Type":"ContainerStarted","Data":"762f3afb0534c3d52306f1387682696329a2cf6c9a7ba82f5f0aa20c237fce74"}
Sep 16 21:33:56.673613 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:56.673589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d" event={"ID":"c09e02b9-d226-419a-8a83-89efcd4825fc","Type":"ContainerStarted","Data":"dbe87a25bfccf494935ae45ada54facb48b8e66548430b4346d0272e7c1157fd"}
Sep 16 21:33:56.673767 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:56.673628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d" event={"ID":"c09e02b9-d226-419a-8a83-89efcd4825fc","Type":"ContainerStarted","Data":"e7c98fe4936224edd18a5fd7952e92120f7c4ab6d1c233eaef32e8860b8f4248"}
Sep 16 21:33:56.693732 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:56.693671    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr" podStartSLOduration=1.693631931 podStartE2EDuration="1.693631931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:33:56.691397615 +0000 UTC m=+696.591607619" watchObservedRunningTime="2026-09-16 21:33:56.693631931 +0000 UTC m=+696.593841938"
Sep 16 21:33:56.712666 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:56.712602    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m" podStartSLOduration=1.712584611 podStartE2EDuration="1.712584611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:33:56.71070522 +0000 UTC m=+696.610915221" watchObservedRunningTime="2026-09-16 21:33:56.712584611 +0000 UTC m=+696.612794615"
Sep 16 21:33:56.727584 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:56.727548    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:56.729332 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:56.729284    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d" podStartSLOduration=1.7292691059999998 podStartE2EDuration="1.729269106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:33:56.727503233 +0000 UTC m=+696.627713238" watchObservedRunningTime="2026-09-16 21:33:56.729269106 +0000 UTC m=+696.629479107"
Sep 16 21:33:57.464006 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:57.463967    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:57.469022 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:57.468996    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:57.599017 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:57.598977    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:57.604065 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:57.604040    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:57.678724 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:57.678689    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:57.679206 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:57.678754    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:57.679322 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:57.679302    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:33:57.679401 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:57.679386    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:33:57.734158 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:57.734077    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:58.682701 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:58.682646    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:33:58.683437 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:33:58.683420    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:34:03.402092 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.402054    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"]
Sep 16 21:34:03.406117 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.406096    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.409070 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.409046    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-ddbefa-openshift-default-dockercfg-n28wx\""
Sep 16 21:34:03.427865 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.427840    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"]
Sep 16 21:34:03.576375 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.576337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb4c011-6706-4815-83a0-8bccd981be05-istiod-ca-cert\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.576533 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.576378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-workload-certs\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.576533 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.576437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-workload-socket\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.576533 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.576493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76jfs\" (UniqueName: \"kubernetes.io/projected/ebb4c011-6706-4815-83a0-8bccd981be05-kube-api-access-76jfs\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.576533 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.576524    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-istio-data\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.576731 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.576571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb4c011-6706-4815-83a0-8bccd981be05-istio-token\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.576731 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.576610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb4c011-6706-4815-83a0-8bccd981be05-istio-podinfo\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.576731 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.576639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-credential-socket\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.576731 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.576684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-istio-envoy\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.678010 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.677976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb4c011-6706-4815-83a0-8bccd981be05-istio-podinfo\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.678172 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.678022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-credential-socket\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.678172 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.678042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-istio-envoy\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.678172 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.678127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb4c011-6706-4815-83a0-8bccd981be05-istiod-ca-cert\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.678294 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.678163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-workload-certs\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.678294 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.678262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-workload-socket\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.678385 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.678315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76jfs\" (UniqueName: \"kubernetes.io/projected/ebb4c011-6706-4815-83a0-8bccd981be05-kube-api-access-76jfs\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.678385 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.678360    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-istio-data\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.678487 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.678388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb4c011-6706-4815-83a0-8bccd981be05-istio-token\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.678487 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.678472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-credential-socket\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.678923 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.678739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-workload-certs\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.678923 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.678874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-workload-socket\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.678923 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.678902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-istio-data\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.679181 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.679149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb4c011-6706-4815-83a0-8bccd981be05-istiod-ca-cert\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.680365 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.680346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-istio-envoy\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.680573 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.680556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb4c011-6706-4815-83a0-8bccd981be05-istio-podinfo\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.692928 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.692887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb4c011-6706-4815-83a0-8bccd981be05-istio-token\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.699407 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.699377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76jfs\" (UniqueName: \"kubernetes.io/projected/ebb4c011-6706-4815-83a0-8bccd981be05-kube-api-access-76jfs\") pod \"e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") " pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:03.716818 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:03.716791    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:04.015014 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:04.014961    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"]
Sep 16 21:34:04.016315 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:34:04.016253    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb4c011_6706_4815_83a0_8bccd981be05.slice/crio-362b9824665c1c1579fd2be1dad7c544fc176c5f33f6ebae976d31594c22ba05 WatchSource:0}: Error finding container 362b9824665c1c1579fd2be1dad7c544fc176c5f33f6ebae976d31594c22ba05: Status 404 returned error can't find the container with id 362b9824665c1c1579fd2be1dad7c544fc176c5f33f6ebae976d31594c22ba05
Sep 16 21:34:04.018918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:04.018882    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:04.019012 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:04.018949    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:04.019012 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:04.018976    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:04.722465 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:04.722425    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x" event={"ID":"ebb4c011-6706-4815-83a0-8bccd981be05","Type":"ContainerStarted","Data":"37f265530fc93bb27736d93122b9aec35b0dffa9c2af91d1e8d23e1d0db75ae6"}
Sep 16 21:34:04.722951 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:04.722509    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x" event={"ID":"ebb4c011-6706-4815-83a0-8bccd981be05","Type":"ContainerStarted","Data":"362b9824665c1c1579fd2be1dad7c544fc176c5f33f6ebae976d31594c22ba05"}
Sep 16 21:34:04.759205 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:04.759146    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x" podStartSLOduration=1.759129661 podStartE2EDuration="1.759129661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:04.756198667 +0000 UTC m=+704.656408671" watchObservedRunningTime="2026-09-16 21:34:04.759129661 +0000 UTC m=+704.659339666"
Sep 16 21:34:05.717597 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:05.717565    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:05.722587 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:05.722557    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:05.726623 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:05.726600    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:05.727398 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:05.727381    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:06.824344 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:06.824316    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"]
Sep 16 21:34:06.827833 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:06.827816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:06.830296 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:06.830270    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0e078853-openshift-default-dockercfg-ltwr6\""
Sep 16 21:34:06.847314 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:06.847290    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"]
Sep 16 21:34:06.910448 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:06.910412    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2464b0a3-4e63-48b7-b775-de355065a269-istiod-ca-cert\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:06.910746 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:06.910461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2464b0a3-4e63-48b7-b775-de355065a269-istio-podinfo\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:06.910746 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:06.910552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-workload-socket\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:06.910746 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:06.910577    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-credential-socket\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:06.910746 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:06.910606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2464b0a3-4e63-48b7-b775-de355065a269-istio-token\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:06.910746 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:06.910623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-istio-data\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:06.910746 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:06.910639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-workload-certs\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:06.910746 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:06.910682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-istio-envoy\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:06.910746 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:06.910698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z26m\" (UniqueName: \"kubernetes.io/projected/2464b0a3-4e63-48b7-b775-de355065a269-kube-api-access-2z26m\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.011736 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.011701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-workload-socket\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.011921 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.011750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-credential-socket\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.011921 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.011794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2464b0a3-4e63-48b7-b775-de355065a269-istio-token\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.011921 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.011823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-istio-data\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.012112 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.011952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-workload-certs\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.012112 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.011992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-istio-envoy\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.012112 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.012018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z26m\" (UniqueName: \"kubernetes.io/projected/2464b0a3-4e63-48b7-b775-de355065a269-kube-api-access-2z26m\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.012112 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.012098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2464b0a3-4e63-48b7-b775-de355065a269-istiod-ca-cert\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.012309 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.012181    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-workload-socket\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.012309 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.012267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-istio-data\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.012383 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.012322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-workload-certs\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.012383 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.012335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-credential-socket\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.012523 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.012500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2464b0a3-4e63-48b7-b775-de355065a269-istio-podinfo\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.012944 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.012924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2464b0a3-4e63-48b7-b775-de355065a269-istiod-ca-cert\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.014428 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.014401    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-istio-envoy\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.014877 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.014859    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2464b0a3-4e63-48b7-b775-de355065a269-istio-podinfo\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.040280 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.040210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z26m\" (UniqueName: \"kubernetes.io/projected/2464b0a3-4e63-48b7-b775-de355065a269-kube-api-access-2z26m\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.053125 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.053058    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2464b0a3-4e63-48b7-b775-de355065a269-istio-token\") pod \"e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") " pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.139127 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.139026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:07.430916 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.430874    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"]
Sep 16 21:34:07.444519 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.444443    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:07.444695 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.444533    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:07.444695 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.444578    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:07.740224 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.740128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m" event={"ID":"2464b0a3-4e63-48b7-b775-de355065a269","Type":"ContainerStarted","Data":"09c464bc5ac5c4e8a2f11368898c168705d1c5bc6ecb5b0d49be3c319a5b9a42"}
Sep 16 21:34:07.740462 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.740441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m" event={"ID":"2464b0a3-4e63-48b7-b775-de355065a269","Type":"ContainerStarted","Data":"29a9143416cba4e9415a6d66d040d9bcc6cd9b22f983c67b56d0aa84a8676cc1"}
Sep 16 21:34:07.769072 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:07.769008    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m" podStartSLOduration=1.768983493 podStartE2EDuration="1.768983493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:07.766887797 +0000 UTC m=+707.667097803" watchObservedRunningTime="2026-09-16 21:34:07.768983493 +0000 UTC m=+707.669193498"
Sep 16 21:34:08.140330 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:08.140253    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:08.142087 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:08.142064    2563 patch_prober.go:28] interesting pod/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 16 21:34:08.142227 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:08.142207    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m" podUID="2464b0a3-4e63-48b7-b775-de355065a269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 16 21:34:09.184198 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:09.183947    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:09.754644 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:09.754606    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:09.756245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:09.756223    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:14.334204 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.334102    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"]
Sep 16 21:34:14.338854 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.338831    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.343477 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.343306    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-44e517-openshift-default-dockercfg-pxwq8\""
Sep 16 21:34:14.396936 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.396904    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"]
Sep 16 21:34:14.434793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.434248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcz6z\" (UniqueName: \"kubernetes.io/projected/83813d1b-be84-4dbc-a6a3-8f8f2f850486-kube-api-access-lcz6z\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.434793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.434317    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-data\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.434793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.434387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-workload-certs\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.434793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.434411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-envoy\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.434793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.434441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-credential-socket\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.434793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.434471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-token\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.434793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.434501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-podinfo\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.434793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.434596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istiod-ca-cert\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.434793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.434642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-workload-socket\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.536212 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.536123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-data\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.536212 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.536190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-workload-certs\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.536437 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.536216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-envoy\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.536437 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.536244    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-credential-socket\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.536437 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.536276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-token\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.536437 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.536302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-podinfo\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.536437 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.536383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istiod-ca-cert\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.536437 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.536423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-workload-socket\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.536758 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.536487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcz6z\" (UniqueName: \"kubernetes.io/projected/83813d1b-be84-4dbc-a6a3-8f8f2f850486-kube-api-access-lcz6z\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.536758 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.536571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-data\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.536758 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.536645    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-workload-certs\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.537440 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.537035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-credential-socket\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.537440 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.537325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-workload-socket\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.537744 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.537724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istiod-ca-cert\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.539584 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.539523    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-envoy\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.539860 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.539823    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-podinfo\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.554059 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.553758    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcz6z\" (UniqueName: \"kubernetes.io/projected/83813d1b-be84-4dbc-a6a3-8f8f2f850486-kube-api-access-lcz6z\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.560649 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.560619    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-token\") pod \"e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") " pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.653305 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.653218    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:14.724959 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.724924    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 21:34:14.725116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.724981    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m" podUID="7f966d85-1c38-4502-bc7b-d36d78985f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 21:34:14.971142 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.970800    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"]
Sep 16 21:34:14.974235 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:34:14.974179    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83813d1b_be84_4dbc_a6a3_8f8f2f850486.slice/crio-5b9be0a43f1f2c13df093dfa9b60221464124783758c030412cd3d452c56bb2e WatchSource:0}: Error finding container 5b9be0a43f1f2c13df093dfa9b60221464124783758c030412cd3d452c56bb2e: Status 404 returned error can't find the container with id 5b9be0a43f1f2c13df093dfa9b60221464124783758c030412cd3d452c56bb2e
Sep 16 21:34:14.976960 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.976918    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:14.977071 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.977013    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:14.977071 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:14.977059    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:15.794944 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:15.794892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42" event={"ID":"83813d1b-be84-4dbc-a6a3-8f8f2f850486","Type":"ContainerStarted","Data":"96ea0f757e4b4fc9734b24c7d23f515689ad30afc253d29744c6e762f9c0d7e0"}
Sep 16 21:34:15.795424 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:15.794969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42" event={"ID":"83813d1b-be84-4dbc-a6a3-8f8f2f850486","Type":"ContainerStarted","Data":"5b9be0a43f1f2c13df093dfa9b60221464124783758c030412cd3d452c56bb2e"}
Sep 16 21:34:15.863792 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:15.863662    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42" podStartSLOduration=1.8636348520000001 podStartE2EDuration="1.863634852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:15.859839535 +0000 UTC m=+715.760049565" watchObservedRunningTime="2026-09-16 21:34:15.863634852 +0000 UTC m=+715.763844857"
Sep 16 21:34:16.653875 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:16.653831    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:16.659240 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:16.659211    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:16.806774 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:16.806509    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:16.808165 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:16.808134    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:18.906194 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:34:18.906121    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f966d85_1c38_4502_bc7b_d36d78985f58.slice/crio-conmon-5dbad252c3a97baa344fc3cf5187ed474fa2dd7cd1ba717e3cb57e85ef212422.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@2-13429-0.service\": RecentStats: unable to find data in memory cache]"
Sep 16 21:34:19.823607 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.823572    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f966d85-1c38-4502-bc7b-d36d78985f58" containerID="5dbad252c3a97baa344fc3cf5187ed474fa2dd7cd1ba717e3cb57e85ef212422" exitCode=0
Sep 16 21:34:19.823821 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.823677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m" event={"ID":"7f966d85-1c38-4502-bc7b-d36d78985f58","Type":"ContainerDied","Data":"5dbad252c3a97baa344fc3cf5187ed474fa2dd7cd1ba717e3cb57e85ef212422"}
Sep 16 21:34:19.824203 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.824179    2563 scope.go:117] "RemoveContainer" containerID="5dbad252c3a97baa344fc3cf5187ed474fa2dd7cd1ba717e3cb57e85ef212422"
Sep 16 21:34:19.825414 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.825379    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:19.825414 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.825407    2563 generic.go:356] "Generic (PLEG): container finished" podID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerID="dbe87a25bfccf494935ae45ada54facb48b8e66548430b4346d0272e7c1157fd" exitCode=0
Sep 16 21:34:19.825579 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.825450    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:19.825579 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.825482    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:19.825579 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.825491    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d" event={"ID":"c09e02b9-d226-419a-8a83-89efcd4825fc","Type":"ContainerDied","Data":"dbe87a25bfccf494935ae45ada54facb48b8e66548430b4346d0272e7c1157fd"}
Sep 16 21:34:19.825951 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.825932    2563 scope.go:117] "RemoveContainer" containerID="dbe87a25bfccf494935ae45ada54facb48b8e66548430b4346d0272e7c1157fd"
Sep 16 21:34:19.827123 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.827105    2563 generic.go:356] "Generic (PLEG): container finished" podID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerID="c2095f774fd1e1360787e43484333f46f8b507dc8a6bf21dca4195453c28ae0d" exitCode=0
Sep 16 21:34:19.827198 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.827122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr" event={"ID":"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8","Type":"ContainerDied","Data":"c2095f774fd1e1360787e43484333f46f8b507dc8a6bf21dca4195453c28ae0d"}
Sep 16 21:34:19.827198 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.827101    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:19.827198 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.827186    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:19.827313 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.827223    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:19.827435 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.827423    2563 scope.go:117] "RemoveContainer" containerID="c2095f774fd1e1360787e43484333f46f8b507dc8a6bf21dca4195453c28ae0d"
Sep 16 21:34:19.828522 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.828442    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:19.828522 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.828491    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:19.828522 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:19.828520    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:20.843994 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:20.843862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m" event={"ID":"7f966d85-1c38-4502-bc7b-d36d78985f58","Type":"ContainerStarted","Data":"3515b914829d4880738c1143f53310f003e0622b186475a88fcb9bab80753f96"}
Sep 16 21:34:20.846035 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:20.846011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d" event={"ID":"c09e02b9-d226-419a-8a83-89efcd4825fc","Type":"ContainerStarted","Data":"2332f673de92c63b590901419b464abfb6daadc53fc277d238b6e87b7cdc5cde"}
Sep 16 21:34:20.848158 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:20.848138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr" event={"ID":"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8","Type":"ContainerStarted","Data":"226ecd91c09196b414f3ce34c95c835a5f788617d40dfcbad19d70eaedf790f3"}
Sep 16 21:34:21.464033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:21.463995    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:34:21.464214 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:21.464124    2563 patch_prober.go:28] interesting pod/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 16 21:34:21.464214 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:21.464170    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr" podUID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 16 21:34:21.598916 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:21.598875    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:34:21.599125 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:21.599008    2563 patch_prober.go:28] interesting pod/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 16 21:34:21.599125 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:21.599052    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d" podUID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 16 21:34:21.728815 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:21.728715    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:34:21.728991 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:21.728968    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 16 21:34:21.729062 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:21.729044    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m" podUID="7f966d85-1c38-4502-bc7b-d36d78985f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 16 21:34:22.463929 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:22.463887    2563 patch_prober.go:28] interesting pod/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 16 21:34:22.464273 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:22.463961    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr" podUID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 16 21:34:22.599907 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:22.599419    2563 patch_prober.go:28] interesting pod/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 16 21:34:22.599907 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:22.599486    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d" podUID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 16 21:34:22.728256 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:22.728151    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 16 21:34:22.728256 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:22.728232    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m" podUID="7f966d85-1c38-4502-bc7b-d36d78985f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 16 21:34:23.463452 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:23.463415    2563 patch_prober.go:28] interesting pod/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 16 21:34:23.463622 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:23.463466    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr" podUID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 16 21:34:23.599674 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:23.599612    2563 patch_prober.go:28] interesting pod/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 16 21:34:23.600147 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:23.599702    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d" podUID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 16 21:34:23.728281 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:23.728180    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 16 21:34:23.728281 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:23.728264    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m" podUID="7f966d85-1c38-4502-bc7b-d36d78985f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 16 21:34:24.464175 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:24.464129    2563 patch_prober.go:28] interesting pod/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 16 21:34:24.464383 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:24.464203    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr" podUID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 16 21:34:24.599342 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:24.599297    2563 patch_prober.go:28] interesting pod/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 16 21:34:24.599543 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:24.599368    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d" podUID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 16 21:34:24.728737 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:24.727921    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 16 21:34:24.728737 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:24.728011    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m" podUID="7f966d85-1c38-4502-bc7b-d36d78985f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 16 21:34:25.464332 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:25.464288    2563 patch_prober.go:28] interesting pod/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 16 21:34:25.464585 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:25.464372    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr" podUID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 16 21:34:25.599680 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:25.599069    2563 patch_prober.go:28] interesting pod/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 16 21:34:25.599680 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:25.599152    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d" podUID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 16 21:34:25.730758 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:25.730057    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 16 21:34:25.730758 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:25.730142    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m" podUID="7f966d85-1c38-4502-bc7b-d36d78985f58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 16 21:34:26.464171 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:26.464129    2563 patch_prober.go:28] interesting pod/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 16 21:34:26.464466 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:26.464433    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr" podUID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 16 21:34:26.599786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:26.599336    2563 patch_prober.go:28] interesting pod/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 16 21:34:26.599786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:26.599398    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d" podUID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 16 21:34:26.838450 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:26.836358    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:34:26.838450 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:26.838213    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:34:26.839517 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:26.839466    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:34:27.464178 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:27.464144    2563 patch_prober.go:28] interesting pod/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Sep 16 21:34:27.464330 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:27.464204    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr" podUID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Sep 16 21:34:27.616172 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:27.616142    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:34:27.616634 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:27.616606    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:34:27.618044 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:27.618022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:34:27.678729 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:27.678698    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:34:28.474292 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:28.472879    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:34:28.480254 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:28.480220    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:34:28.913482 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:28.913394    2563 generic.go:356] "Generic (PLEG): container finished" podID="ebb4c011-6706-4815-83a0-8bccd981be05" containerID="37f265530fc93bb27736d93122b9aec35b0dffa9c2af91d1e8d23e1d0db75ae6" exitCode=0
Sep 16 21:34:28.913482 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:28.913468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x" event={"ID":"ebb4c011-6706-4815-83a0-8bccd981be05","Type":"ContainerDied","Data":"37f265530fc93bb27736d93122b9aec35b0dffa9c2af91d1e8d23e1d0db75ae6"}
Sep 16 21:34:28.914068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:28.914052    2563 scope.go:117] "RemoveContainer" containerID="37f265530fc93bb27736d93122b9aec35b0dffa9c2af91d1e8d23e1d0db75ae6"
Sep 16 21:34:28.916348 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:28.915759    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:28.916348 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:28.915928    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:28.916348 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:28.916024    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:29.690296 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:29.690234    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx"]
Sep 16 21:34:29.694547 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:29.694515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx"
Sep 16 21:34:29.701022 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:29.700979    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:34:29.701402 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:29.701383    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:34:29.708553 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:29.708528    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx"]
Sep 16 21:34:29.854069 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:29.854026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b3972e-9a3f-4140-8c6b-48ab1350e71f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e078853-25tlx\" (UID: \"65b3972e-9a3f-4140-8c6b-48ab1350e71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx"
Sep 16 21:34:29.920846 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:29.920814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x" event={"ID":"ebb4c011-6706-4815-83a0-8bccd981be05","Type":"ContainerStarted","Data":"cc11bddc036aa43df220623d56ad5744bbdc77eb65118ff77224df5dfe86a9d5"}
Sep 16 21:34:29.954963 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:29.954877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b3972e-9a3f-4140-8c6b-48ab1350e71f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e078853-25tlx\" (UID: \"65b3972e-9a3f-4140-8c6b-48ab1350e71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx"
Sep 16 21:34:29.955524 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:29.955499    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b3972e-9a3f-4140-8c6b-48ab1350e71f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e078853-25tlx\" (UID: \"65b3972e-9a3f-4140-8c6b-48ab1350e71f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx"
Sep 16 21:34:30.006015 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:30.005977    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx"
Sep 16 21:34:30.288742 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:30.287502    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx"]
Sep 16 21:34:30.303677 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:34:30.302955    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b3972e_9a3f_4140_8c6b_48ab1350e71f.slice/crio-c7db22409ea3c3eb63ffae4b292cbffd94b076413774b114ec84cc7fcaf494fe WatchSource:0}: Error finding container c7db22409ea3c3eb63ffae4b292cbffd94b076413774b114ec84cc7fcaf494fe: Status 404 returned error can't find the container with id c7db22409ea3c3eb63ffae4b292cbffd94b076413774b114ec84cc7fcaf494fe
Sep 16 21:34:30.717600 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:30.717569    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:30.717978 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:30.717872    2563 patch_prober.go:28] interesting pod/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 16 21:34:30.717978 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:30.717929    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x" podUID="ebb4c011-6706-4815-83a0-8bccd981be05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 16 21:34:30.927380 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:30.927338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx" event={"ID":"65b3972e-9a3f-4140-8c6b-48ab1350e71f","Type":"ContainerStarted","Data":"c7db22409ea3c3eb63ffae4b292cbffd94b076413774b114ec84cc7fcaf494fe"}
Sep 16 21:34:31.718156 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:31.718110    2563 patch_prober.go:28] interesting pod/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 16 21:34:31.718558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:31.718184    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x" podUID="ebb4c011-6706-4815-83a0-8bccd981be05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 16 21:34:32.718229 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:32.718194    2563 patch_prober.go:28] interesting pod/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 16 21:34:32.718631 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:32.718265    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x" podUID="ebb4c011-6706-4815-83a0-8bccd981be05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 16 21:34:33.717688 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:33.717631    2563 patch_prober.go:28] interesting pod/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 16 21:34:33.717856 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:33.717712    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x" podUID="ebb4c011-6706-4815-83a0-8bccd981be05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 16 21:34:34.718307 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:34.718264    2563 patch_prober.go:28] interesting pod/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 16 21:34:34.720876 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:34.718331    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x" podUID="ebb4c011-6706-4815-83a0-8bccd981be05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 16 21:34:34.950394 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:34.950344    2563 generic.go:356] "Generic (PLEG): container finished" podID="2464b0a3-4e63-48b7-b775-de355065a269" containerID="09c464bc5ac5c4e8a2f11368898c168705d1c5bc6ecb5b0d49be3c319a5b9a42" exitCode=0
Sep 16 21:34:34.950578 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:34.950421    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m" event={"ID":"2464b0a3-4e63-48b7-b775-de355065a269","Type":"ContainerDied","Data":"09c464bc5ac5c4e8a2f11368898c168705d1c5bc6ecb5b0d49be3c319a5b9a42"}
Sep 16 21:34:34.950882 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:34.950868    2563 scope.go:117] "RemoveContainer" containerID="09c464bc5ac5c4e8a2f11368898c168705d1c5bc6ecb5b0d49be3c319a5b9a42"
Sep 16 21:34:34.952006 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:34.951978    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:34.952073 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:34.952051    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:34.952110 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:34.952082    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:35.718046 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:35.717578    2563 patch_prober.go:28] interesting pod/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 16 21:34:35.718046 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:35.717647    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x" podUID="ebb4c011-6706-4815-83a0-8bccd981be05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 16 21:34:35.727762 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:35.726881    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:35.958746 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:35.958709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m" event={"ID":"2464b0a3-4e63-48b7-b775-de355065a269","Type":"ContainerStarted","Data":"7903e585436a4654f0257dc1115160d62cceb3d1afbd9b2ccf0a8c58092a5a50"}
Sep 16 21:34:36.139883 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:36.139792    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:36.141119 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:36.141091    2563 patch_prober.go:28] interesting pod/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 16 21:34:36.141234 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:36.141143    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m" podUID="2464b0a3-4e63-48b7-b775-de355065a269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 16 21:34:36.722405 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:36.722366    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:36.723709 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:36.723688    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:34:37.140001 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:37.139915    2563 patch_prober.go:28] interesting pod/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 16 21:34:37.140001 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:37.139972    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m" podUID="2464b0a3-4e63-48b7-b775-de355065a269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 16 21:34:38.140288 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:38.140250    2563 patch_prober.go:28] interesting pod/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 16 21:34:38.140691 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:38.140311    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m" podUID="2464b0a3-4e63-48b7-b775-de355065a269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 16 21:34:39.140105 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:39.140064    2563 patch_prober.go:28] interesting pod/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 16 21:34:39.140287 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:39.140133    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m" podUID="2464b0a3-4e63-48b7-b775-de355065a269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 16 21:34:39.754492 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:39.754461    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:40.140107 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:40.140075    2563 patch_prober.go:28] interesting pod/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 16 21:34:40.140279 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:40.140145    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m" podUID="2464b0a3-4e63-48b7-b775-de355065a269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 16 21:34:41.139483 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:41.139436    2563 patch_prober.go:28] interesting pod/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 16 21:34:41.139888 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:41.139509    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m" podUID="2464b0a3-4e63-48b7-b775-de355065a269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 16 21:34:42.144420 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:42.144384    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:42.145720 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:42.145695    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:34:43.014093 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:43.014015    2563 generic.go:356] "Generic (PLEG): container finished" podID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" containerID="96ea0f757e4b4fc9734b24c7d23f515689ad30afc253d29744c6e762f9c0d7e0" exitCode=0
Sep 16 21:34:43.014273 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:43.014091    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42" event={"ID":"83813d1b-be84-4dbc-a6a3-8f8f2f850486","Type":"ContainerDied","Data":"96ea0f757e4b4fc9734b24c7d23f515689ad30afc253d29744c6e762f9c0d7e0"}
Sep 16 21:34:43.014634 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:43.014619    2563 scope.go:117] "RemoveContainer" containerID="96ea0f757e4b4fc9734b24c7d23f515689ad30afc253d29744c6e762f9c0d7e0"
Sep 16 21:34:43.015730 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:43.015701    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:43.015792 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:43.015771    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:43.015827 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:43.015804    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:34:44.020636 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:44.020606    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42" event={"ID":"83813d1b-be84-4dbc-a6a3-8f8f2f850486","Type":"ContainerStarted","Data":"16e1b5981f429adc87691fba1bab4d3ff5125d0ffb54d25e9d68adb0da3dcf11"}
Sep 16 21:34:44.653973 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:44.653942    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:44.655105 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:44.655078    2563 patch_prober.go:28] interesting pod/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 16 21:34:44.655226 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:44.655137    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42" podUID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 16 21:34:45.654320 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:45.654288    2563 patch_prober.go:28] interesting pod/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 16 21:34:45.654782 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:45.654349    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42" podUID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 16 21:34:46.654027 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:46.653987    2563 patch_prober.go:28] interesting pod/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 16 21:34:46.654191 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:46.654047    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42" podUID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 16 21:34:46.806644 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:46.806555    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:47.037574 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:47.037539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx" event={"ID":"65b3972e-9a3f-4140-8c6b-48ab1350e71f","Type":"ContainerStarted","Data":"dd35502c37084fd6c2d6f4ac266e74b3b5f26dbfc87b3f4f25dfb693bbf24737"}
Sep 16 21:34:47.057396 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:47.057292    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx" podStartSLOduration=1.901450191 podStartE2EDuration="18.057277077s" totalImagesPullingTime="16.155826886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:47.055719908 +0000 UTC m=+746.955929911" watchObservedRunningTime="2026-09-16 21:34:47.057277077 +0000 UTC m=+746.957487081"
Sep 16 21:34:47.654321 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:47.654289    2563 patch_prober.go:28] interesting pod/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 16 21:34:47.654501 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:47.654346    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42" podUID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 16 21:34:48.044109 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:48.044078    2563 generic.go:356] "Generic (PLEG): container finished" podID="65b3972e-9a3f-4140-8c6b-48ab1350e71f" containerID="dd35502c37084fd6c2d6f4ac266e74b3b5f26dbfc87b3f4f25dfb693bbf24737" exitCode=0
Sep 16 21:34:48.044496 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:48.044121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx" event={"ID":"65b3972e-9a3f-4140-8c6b-48ab1350e71f","Type":"ContainerDied","Data":"dd35502c37084fd6c2d6f4ac266e74b3b5f26dbfc87b3f4f25dfb693bbf24737"}
Sep 16 21:34:48.653623 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:48.653587    2563 patch_prober.go:28] interesting pod/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 16 21:34:48.653811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:48.653646    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42" podUID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 16 21:34:49.218097 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:49.218069    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx"
Sep 16 21:34:49.291724 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:49.282134    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b3972e-9a3f-4140-8c6b-48ab1350e71f-maas-api-service-ca\") pod \"65b3972e-9a3f-4140-8c6b-48ab1350e71f\" (UID: \"65b3972e-9a3f-4140-8c6b-48ab1350e71f\") "
Sep 16 21:34:49.291724 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:49.290012    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b3972e-9a3f-4140-8c6b-48ab1350e71f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "65b3972e-9a3f-4140-8c6b-48ab1350e71f" (UID: "65b3972e-9a3f-4140-8c6b-48ab1350e71f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:34:49.383462 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:49.383347    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b3972e-9a3f-4140-8c6b-48ab1350e71f-maas-api-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:34:49.654283 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:49.654192    2563 patch_prober.go:28] interesting pod/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 16 21:34:49.654283 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:49.654248    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42" podUID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 16 21:34:50.057504 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:50.057476    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx"
Sep 16 21:34:50.057703 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:50.057465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx" event={"ID":"65b3972e-9a3f-4140-8c6b-48ab1350e71f","Type":"ContainerDied","Data":"c7db22409ea3c3eb63ffae4b292cbffd94b076413774b114ec84cc7fcaf494fe"}
Sep 16 21:34:50.057703 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:50.057606    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7db22409ea3c3eb63ffae4b292cbffd94b076413774b114ec84cc7fcaf494fe"
Sep 16 21:34:50.657717 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:50.657688    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:34:50.658646 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:34:50.658629    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:35:00.582170 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:00.582129    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx"]
Sep 16 21:35:00.584408 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:00.584379    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e078853-25tlx"]
Sep 16 21:35:00.614446 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:00.614401    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65b3972e-9a3f-4140-8c6b-48ab1350e71f" path="/var/lib/kubelet/pods/65b3972e-9a3f-4140-8c6b-48ab1350e71f/volumes"
Sep 16 21:35:00.720545 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:00.720514    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"]
Sep 16 21:35:00.720900 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:00.720870    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m" podUID="2464b0a3-4e63-48b7-b775-de355065a269" containerName="istio-proxy" containerID="cri-o://7903e585436a4654f0257dc1115160d62cceb3d1afbd9b2ccf0a8c58092a5a50" gracePeriod=30
Sep 16 21:35:01.712461 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.712426    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"]
Sep 16 21:35:01.713008 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.712989    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b3972e-9a3f-4140-8c6b-48ab1350e71f" containerName="revoke-keys"
Sep 16 21:35:01.713094 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.713011    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b3972e-9a3f-4140-8c6b-48ab1350e71f" containerName="revoke-keys"
Sep 16 21:35:01.713148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.713123    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b3972e-9a3f-4140-8c6b-48ab1350e71f" containerName="revoke-keys"
Sep 16 21:35:01.716537 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.716516    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.719066 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.719046    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-595d23dd-gw-openshift-default-dockercfg-qcmf8\""
Sep 16 21:35:01.727067 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.727046    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"]
Sep 16 21:35:01.819261 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.819223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-data\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.819515 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.819488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8h6x\" (UniqueName: \"kubernetes.io/projected/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-kube-api-access-g8h6x\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.819689 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.819649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-workload-certs\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.819819 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.819804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istiod-ca-cert\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.819941 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.819928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-podinfo\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.820054 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.820038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-workload-socket\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.820217 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.820203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-credential-socket\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.820378 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.820364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-token\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.820515 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.820498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-envoy\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.921608 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.921573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-token\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.921878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.921623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-envoy\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.921878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.921668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-data\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.921878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.921695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8h6x\" (UniqueName: \"kubernetes.io/projected/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-kube-api-access-g8h6x\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.921878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.921722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-workload-certs\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.921878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.921749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istiod-ca-cert\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.921878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.921779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-podinfo\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.921878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.921814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-workload-socket\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.922258 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.921896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-credential-socket\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.922258 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.922165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-data\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.922373 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.922261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-credential-socket\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.922373 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.922299    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-workload-certs\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.922373 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.922334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-workload-socket\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.922637 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.922612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istiod-ca-cert\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.924112 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.924090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-envoy\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.924291 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.924274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-podinfo\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.929539 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.929517    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-token\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:01.929635 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:01.929622    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8h6x\" (UniqueName: \"kubernetes.io/projected/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-kube-api-access-g8h6x\") pod \"e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") " pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:02.027630 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:02.027547    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:02.307110 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:02.306996    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"]
Sep 16 21:35:02.309354 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:35:02.309322    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd149156_3e03_4cf3_b77d_6e9da1fb11bc.slice/crio-dfdc1df8f5eeaece2acf609b965560184b9d6c4d06d27fed0c1b9c128cc65ab5 WatchSource:0}: Error finding container dfdc1df8f5eeaece2acf609b965560184b9d6c4d06d27fed0c1b9c128cc65ab5: Status 404 returned error can't find the container with id dfdc1df8f5eeaece2acf609b965560184b9d6c4d06d27fed0c1b9c128cc65ab5
Sep 16 21:35:02.311980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:02.311936    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:35:02.312096 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:02.312029    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:35:02.312096 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:02.312071    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:35:03.125158 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:03.125127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq" event={"ID":"cd149156-3e03-4cf3-b77d-6e9da1fb11bc","Type":"ContainerStarted","Data":"eb6cc25649498ab4e2a3d0c84d7ec0455a10abea10a271b2ee9e2eb148d0c54b"}
Sep 16 21:35:03.125556 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:03.125176    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq" event={"ID":"cd149156-3e03-4cf3-b77d-6e9da1fb11bc","Type":"ContainerStarted","Data":"dfdc1df8f5eeaece2acf609b965560184b9d6c4d06d27fed0c1b9c128cc65ab5"}
Sep 16 21:35:03.145996 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:03.145932    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq" podStartSLOduration=2.14590088 podStartE2EDuration="2.14590088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:35:03.144158409 +0000 UTC m=+763.044368413" watchObservedRunningTime="2026-09-16 21:35:03.14590088 +0000 UTC m=+763.046110883"
Sep 16 21:35:04.028212 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:04.028177    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:04.033156 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:04.033132    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:04.131242 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:04.131209    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:04.132202 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:04.132182    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:05.971274 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:05.971247    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:35:06.064073 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.063990    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-istio-envoy\") pod \"2464b0a3-4e63-48b7-b775-de355065a269\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") "
Sep 16 21:35:06.064073 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.064072    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-istio-data\") pod \"2464b0a3-4e63-48b7-b775-de355065a269\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") "
Sep 16 21:35:06.064291 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.064107    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-workload-certs\") pod \"2464b0a3-4e63-48b7-b775-de355065a269\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") "
Sep 16 21:35:06.064291 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.064140    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2464b0a3-4e63-48b7-b775-de355065a269-istio-token\") pod \"2464b0a3-4e63-48b7-b775-de355065a269\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") "
Sep 16 21:35:06.064291 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.064166    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2464b0a3-4e63-48b7-b775-de355065a269-istiod-ca-cert\") pod \"2464b0a3-4e63-48b7-b775-de355065a269\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") "
Sep 16 21:35:06.064291 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.064221    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2464b0a3-4e63-48b7-b775-de355065a269-istio-podinfo\") pod \"2464b0a3-4e63-48b7-b775-de355065a269\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") "
Sep 16 21:35:06.064291 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.064253    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-workload-socket\") pod \"2464b0a3-4e63-48b7-b775-de355065a269\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") "
Sep 16 21:35:06.064558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.064296    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-credential-socket\") pod \"2464b0a3-4e63-48b7-b775-de355065a269\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") "
Sep 16 21:35:06.064558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.064337    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z26m\" (UniqueName: \"kubernetes.io/projected/2464b0a3-4e63-48b7-b775-de355065a269-kube-api-access-2z26m\") pod \"2464b0a3-4e63-48b7-b775-de355065a269\" (UID: \"2464b0a3-4e63-48b7-b775-de355065a269\") "
Sep 16 21:35:06.064558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.064397    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2464b0a3-4e63-48b7-b775-de355065a269" (UID: "2464b0a3-4e63-48b7-b775-de355065a269"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:35:06.064737 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.064636    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-istio-data" (OuterVolumeSpecName: "istio-data") pod "2464b0a3-4e63-48b7-b775-de355065a269" (UID: "2464b0a3-4e63-48b7-b775-de355065a269"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:35:06.064792 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.064765    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2464b0a3-4e63-48b7-b775-de355065a269-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2464b0a3-4e63-48b7-b775-de355065a269" (UID: "2464b0a3-4e63-48b7-b775-de355065a269"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:35:06.064852 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.064764    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2464b0a3-4e63-48b7-b775-de355065a269" (UID: "2464b0a3-4e63-48b7-b775-de355065a269"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:35:06.064852 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.064796    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:06.064852 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.064817    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:06.065010 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.064871    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2464b0a3-4e63-48b7-b775-de355065a269" (UID: "2464b0a3-4e63-48b7-b775-de355065a269"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:35:06.066616 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.066589    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2464b0a3-4e63-48b7-b775-de355065a269-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2464b0a3-4e63-48b7-b775-de355065a269" (UID: "2464b0a3-4e63-48b7-b775-de355065a269"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:35:06.066616 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.066602    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2464b0a3-4e63-48b7-b775-de355065a269-kube-api-access-2z26m" (OuterVolumeSpecName: "kube-api-access-2z26m") pod "2464b0a3-4e63-48b7-b775-de355065a269" (UID: "2464b0a3-4e63-48b7-b775-de355065a269"). InnerVolumeSpecName "kube-api-access-2z26m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:35:06.066813 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.066705    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2464b0a3-4e63-48b7-b775-de355065a269" (UID: "2464b0a3-4e63-48b7-b775-de355065a269"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:35:06.066813 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.066788    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2464b0a3-4e63-48b7-b775-de355065a269-istio-token" (OuterVolumeSpecName: "istio-token") pod "2464b0a3-4e63-48b7-b775-de355065a269" (UID: "2464b0a3-4e63-48b7-b775-de355065a269"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:35:06.144218 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.144184    2563 generic.go:356] "Generic (PLEG): container finished" podID="2464b0a3-4e63-48b7-b775-de355065a269" containerID="7903e585436a4654f0257dc1115160d62cceb3d1afbd9b2ccf0a8c58092a5a50" exitCode=0
Sep 16 21:35:06.144410 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.144247    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"
Sep 16 21:35:06.144410 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.144267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m" event={"ID":"2464b0a3-4e63-48b7-b775-de355065a269","Type":"ContainerDied","Data":"7903e585436a4654f0257dc1115160d62cceb3d1afbd9b2ccf0a8c58092a5a50"}
Sep 16 21:35:06.144410 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.144322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m" event={"ID":"2464b0a3-4e63-48b7-b775-de355065a269","Type":"ContainerDied","Data":"29a9143416cba4e9415a6d66d040d9bcc6cd9b22f983c67b56d0aa84a8676cc1"}
Sep 16 21:35:06.144410 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.144345    2563 scope.go:117] "RemoveContainer" containerID="7903e585436a4654f0257dc1115160d62cceb3d1afbd9b2ccf0a8c58092a5a50"
Sep 16 21:35:06.154124 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.154107    2563 scope.go:117] "RemoveContainer" containerID="09c464bc5ac5c4e8a2f11368898c168705d1c5bc6ecb5b0d49be3c319a5b9a42"
Sep 16 21:35:06.162576 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.162557    2563 scope.go:117] "RemoveContainer" containerID="7903e585436a4654f0257dc1115160d62cceb3d1afbd9b2ccf0a8c58092a5a50"
Sep 16 21:35:06.162865 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:35:06.162845    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7903e585436a4654f0257dc1115160d62cceb3d1afbd9b2ccf0a8c58092a5a50\": container with ID starting with 7903e585436a4654f0257dc1115160d62cceb3d1afbd9b2ccf0a8c58092a5a50 not found: ID does not exist" containerID="7903e585436a4654f0257dc1115160d62cceb3d1afbd9b2ccf0a8c58092a5a50"
Sep 16 21:35:06.162943 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.162879    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7903e585436a4654f0257dc1115160d62cceb3d1afbd9b2ccf0a8c58092a5a50"} err="failed to get container status \"7903e585436a4654f0257dc1115160d62cceb3d1afbd9b2ccf0a8c58092a5a50\": rpc error: code = NotFound desc = could not find container \"7903e585436a4654f0257dc1115160d62cceb3d1afbd9b2ccf0a8c58092a5a50\": container with ID starting with 7903e585436a4654f0257dc1115160d62cceb3d1afbd9b2ccf0a8c58092a5a50 not found: ID does not exist"
Sep 16 21:35:06.162943 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.162905    2563 scope.go:117] "RemoveContainer" containerID="09c464bc5ac5c4e8a2f11368898c168705d1c5bc6ecb5b0d49be3c319a5b9a42"
Sep 16 21:35:06.163177 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:35:06.163161    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c464bc5ac5c4e8a2f11368898c168705d1c5bc6ecb5b0d49be3c319a5b9a42\": container with ID starting with 09c464bc5ac5c4e8a2f11368898c168705d1c5bc6ecb5b0d49be3c319a5b9a42 not found: ID does not exist" containerID="09c464bc5ac5c4e8a2f11368898c168705d1c5bc6ecb5b0d49be3c319a5b9a42"
Sep 16 21:35:06.163217 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.163186    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c464bc5ac5c4e8a2f11368898c168705d1c5bc6ecb5b0d49be3c319a5b9a42"} err="failed to get container status \"09c464bc5ac5c4e8a2f11368898c168705d1c5bc6ecb5b0d49be3c319a5b9a42\": rpc error: code = NotFound desc = could not find container \"09c464bc5ac5c4e8a2f11368898c168705d1c5bc6ecb5b0d49be3c319a5b9a42\": container with ID starting with 09c464bc5ac5c4e8a2f11368898c168705d1c5bc6ecb5b0d49be3c319a5b9a42 not found: ID does not exist"
Sep 16 21:35:06.166042 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.166020    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"]
Sep 16 21:35:06.166132 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.166022    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2464b0a3-4e63-48b7-b775-de355065a269-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:06.166132 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.166071    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2464b0a3-4e63-48b7-b775-de355065a269-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:06.166132 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.166081    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2464b0a3-4e63-48b7-b775-de355065a269-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:06.166132 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.166089    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:06.166132 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.166098    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:06.166132 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.166107    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z26m\" (UniqueName: \"kubernetes.io/projected/2464b0a3-4e63-48b7-b775-de355065a269-kube-api-access-2z26m\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:06.166132 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.166117    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2464b0a3-4e63-48b7-b775-de355065a269-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:06.170379 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.170356    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0e078853-openshift-default-7cf5d46d6d-ql22m"]
Sep 16 21:35:06.608251 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:06.608217    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2464b0a3-4e63-48b7-b775-de355065a269" path="/var/lib/kubelet/pods/2464b0a3-4e63-48b7-b775-de355065a269/volumes"
Sep 16 21:35:19.056335 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.056300    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"]
Sep 16 21:35:19.056806 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.056678    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2464b0a3-4e63-48b7-b775-de355065a269" containerName="istio-proxy"
Sep 16 21:35:19.056806 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.056690    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2464b0a3-4e63-48b7-b775-de355065a269" containerName="istio-proxy"
Sep 16 21:35:19.056806 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.056771    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2464b0a3-4e63-48b7-b775-de355065a269" containerName="istio-proxy"
Sep 16 21:35:19.056806 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.056780    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2464b0a3-4e63-48b7-b775-de355065a269" containerName="istio-proxy"
Sep 16 21:35:19.059516 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.059497    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.062893 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.062866    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e047d8\"/\"openshift-service-ca.crt\""
Sep 16 21:35:19.063092 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.063073    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e047d8\"/\"kube-root-ca.crt\""
Sep 16 21:35:19.064063 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.063131    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e047d8\"/\"default-dockercfg-hxdqm\""
Sep 16 21:35:19.064063 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.063183    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e047d8\"/\"simulator-w5-e047d8-kserve-self-signed-certs\""
Sep 16 21:35:19.072288 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.066105    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"]
Sep 16 21:35:19.099641 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.099601    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-dshm\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.099829 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.099693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-tmp-dir\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.099829 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.099761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0500ae-a666-4950-be56-98850da5f370-tls-certs\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.099985 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.099807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-home\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.099985 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.099941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmglp\" (UniqueName: \"kubernetes.io/projected/3d0500ae-a666-4950-be56-98850da5f370-kube-api-access-gmglp\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.100102 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.099988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-model-cache\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.201234 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.201191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmglp\" (UniqueName: \"kubernetes.io/projected/3d0500ae-a666-4950-be56-98850da5f370-kube-api-access-gmglp\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.201414 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.201242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-model-cache\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.201414 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.201288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-dshm\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.201414 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.201307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-tmp-dir\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.201414 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.201340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0500ae-a666-4950-be56-98850da5f370-tls-certs\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.201414 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.201356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-home\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.201799 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.201753    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-home\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.202216 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.202193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-tmp-dir\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.202338 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.202318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-model-cache\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.203910 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.203889    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-dshm\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.204556 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.204539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0500ae-a666-4950-be56-98850da5f370-tls-certs\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.209608 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.209583    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmglp\" (UniqueName: \"kubernetes.io/projected/3d0500ae-a666-4950-be56-98850da5f370-kube-api-access-gmglp\") pod \"simulator-w5-e047d8-kserve-5778687c78-nt5lp\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") " pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.377372 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.377281    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:19.649469 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:19.649440    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"]
Sep 16 21:35:19.652135 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:35:19.652109    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d0500ae_a666_4950_be56_98850da5f370.slice/crio-35ca525ada548030c4e6353f1b5396302798ef68c6bb7edc1e46ea6d0574437d WatchSource:0}: Error finding container 35ca525ada548030c4e6353f1b5396302798ef68c6bb7edc1e46ea6d0574437d: Status 404 returned error can't find the container with id 35ca525ada548030c4e6353f1b5396302798ef68c6bb7edc1e46ea6d0574437d
Sep 16 21:35:20.220755 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:20.220726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp" event={"ID":"3d0500ae-a666-4950-be56-98850da5f370","Type":"ContainerStarted","Data":"8d600fee723b2f3ee3f5adfc3a569f16082577eb44c081379192d16bdfa96170"}
Sep 16 21:35:20.221139 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:20.220763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp" event={"ID":"3d0500ae-a666-4950-be56-98850da5f370","Type":"ContainerStarted","Data":"35ca525ada548030c4e6353f1b5396302798ef68c6bb7edc1e46ea6d0574437d"}
Sep 16 21:35:20.235492 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:20.235441    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp" podStartSLOduration=1.235426206 podStartE2EDuration="1.235426206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:35:20.234024834 +0000 UTC m=+780.134234838" watchObservedRunningTime="2026-09-16 21:35:20.235426206 +0000 UTC m=+780.135636210"
Sep 16 21:35:26.252395 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:26.252357    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd149156-3e03-4cf3-b77d-6e9da1fb11bc" containerID="eb6cc25649498ab4e2a3d0c84d7ec0455a10abea10a271b2ee9e2eb148d0c54b" exitCode=0
Sep 16 21:35:26.252822 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:26.252420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq" event={"ID":"cd149156-3e03-4cf3-b77d-6e9da1fb11bc","Type":"ContainerDied","Data":"eb6cc25649498ab4e2a3d0c84d7ec0455a10abea10a271b2ee9e2eb148d0c54b"}
Sep 16 21:35:26.252869 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:26.252827    2563 scope.go:117] "RemoveContainer" containerID="eb6cc25649498ab4e2a3d0c84d7ec0455a10abea10a271b2ee9e2eb148d0c54b"
Sep 16 21:35:26.254042 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:26.254012    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:35:26.254107 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:26.254085    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:35:26.254144 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:26.254120    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:35:27.258813 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:27.258779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq" event={"ID":"cd149156-3e03-4cf3-b77d-6e9da1fb11bc","Type":"ContainerStarted","Data":"93be242ff9fd39130032eb1c5149c8cfefc1055f995ca411dd13cd2c938d91cc"}
Sep 16 21:35:28.028275 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:28.028239    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:28.028484 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:28.028463    2563 patch_prober.go:28] interesting pod/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 16 21:35:28.028531 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:28.028511    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq" podUID="cd149156-3e03-4cf3-b77d-6e9da1fb11bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 16 21:35:29.028461 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.028424    2563 patch_prober.go:28] interesting pod/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 16 21:35:29.028879 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.028491    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq" podUID="cd149156-3e03-4cf3-b77d-6e9da1fb11bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 16 21:35:29.377847 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.377742    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:29.378398 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.378373    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:29.409937 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.409901    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:29.546991 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.546958    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx"]
Sep 16 21:35:29.547346 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.547333    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2464b0a3-4e63-48b7-b775-de355065a269" containerName="istio-proxy"
Sep 16 21:35:29.547398 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.547347    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2464b0a3-4e63-48b7-b775-de355065a269" containerName="istio-proxy"
Sep 16 21:35:29.550596 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.550578    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx"
Sep 16 21:35:29.552705 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.552682    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:35:29.552705 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.552684    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:35:29.557495 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.557462    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx"]
Sep 16 21:35:29.611644 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.611610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbf35ee-d22e-4aaf-b3bb-44995dc7397b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx\" (UID: \"ffbf35ee-d22e-4aaf-b3bb-44995dc7397b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx"
Sep 16 21:35:29.712366 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.712322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbf35ee-d22e-4aaf-b3bb-44995dc7397b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx\" (UID: \"ffbf35ee-d22e-4aaf-b3bb-44995dc7397b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx"
Sep 16 21:35:29.713044 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.713026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbf35ee-d22e-4aaf-b3bb-44995dc7397b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx\" (UID: \"ffbf35ee-d22e-4aaf-b3bb-44995dc7397b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx"
Sep 16 21:35:29.861901 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:29.861865    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx"
Sep 16 21:35:30.028807 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:30.028723    2563 patch_prober.go:28] interesting pod/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 16 21:35:30.028807 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:30.028782    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq" podUID="cd149156-3e03-4cf3-b77d-6e9da1fb11bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 16 21:35:30.150009 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:30.149986    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx"]
Sep 16 21:35:30.152462 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:35:30.152438    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffbf35ee_d22e_4aaf_b3bb_44995dc7397b.slice/crio-71ae2f640fe0906b128a8184914082de98fa1a1e69d70688ec40c71ce04ae3b9 WatchSource:0}: Error finding container 71ae2f640fe0906b128a8184914082de98fa1a1e69d70688ec40c71ce04ae3b9: Status 404 returned error can't find the container with id 71ae2f640fe0906b128a8184914082de98fa1a1e69d70688ec40c71ce04ae3b9
Sep 16 21:35:30.292378 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:30.292284    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx" event={"ID":"ffbf35ee-d22e-4aaf-b3bb-44995dc7397b","Type":"ContainerStarted","Data":"71ae2f640fe0906b128a8184914082de98fa1a1e69d70688ec40c71ce04ae3b9"}
Sep 16 21:35:30.303541 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:30.303508    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:35:31.028075 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:31.028041    2563 patch_prober.go:28] interesting pod/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 16 21:35:31.028271 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:31.028094    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq" podUID="cd149156-3e03-4cf3-b77d-6e9da1fb11bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 16 21:35:31.298725 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:31.298627    2563 generic.go:356] "Generic (PLEG): container finished" podID="ffbf35ee-d22e-4aaf-b3bb-44995dc7397b" containerID="896c960b317a6f5e1623c02b49bbce8363e2d797b68d88df775cef32cb896f65" exitCode=0
Sep 16 21:35:31.298725 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:31.298710    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx" event={"ID":"ffbf35ee-d22e-4aaf-b3bb-44995dc7397b","Type":"ContainerDied","Data":"896c960b317a6f5e1623c02b49bbce8363e2d797b68d88df775cef32cb896f65"}
Sep 16 21:35:32.027999 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:32.027963    2563 patch_prober.go:28] interesting pod/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 16 21:35:32.028186 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:32.028017    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq" podUID="cd149156-3e03-4cf3-b77d-6e9da1fb11bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 16 21:35:32.457147 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:32.457119    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx"
Sep 16 21:35:32.541951 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:32.541918    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbf35ee-d22e-4aaf-b3bb-44995dc7397b-maas-api-service-ca\") pod \"ffbf35ee-d22e-4aaf-b3bb-44995dc7397b\" (UID: \"ffbf35ee-d22e-4aaf-b3bb-44995dc7397b\") "
Sep 16 21:35:32.542383 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:32.542351    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffbf35ee-d22e-4aaf-b3bb-44995dc7397b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ffbf35ee-d22e-4aaf-b3bb-44995dc7397b" (UID: "ffbf35ee-d22e-4aaf-b3bb-44995dc7397b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:35:32.643788 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:32.643758    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffbf35ee-d22e-4aaf-b3bb-44995dc7397b-maas-api-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:33.032477 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:33.032444    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:33.032738 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:33.032718    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:33.033373 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:33.033357    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:33.310305 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:33.310214    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx"
Sep 16 21:35:33.310305 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:33.310255    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx" event={"ID":"ffbf35ee-d22e-4aaf-b3bb-44995dc7397b","Type":"ContainerDied","Data":"71ae2f640fe0906b128a8184914082de98fa1a1e69d70688ec40c71ce04ae3b9"}
Sep 16 21:35:33.310504 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:33.310306    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71ae2f640fe0906b128a8184914082de98fa1a1e69d70688ec40c71ce04ae3b9"
Sep 16 21:35:33.989772 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:33.989732    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"]
Sep 16 21:35:33.990141 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:33.990128    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffbf35ee-d22e-4aaf-b3bb-44995dc7397b" containerName="revoke-keys"
Sep 16 21:35:33.990196 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:33.990142    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffbf35ee-d22e-4aaf-b3bb-44995dc7397b" containerName="revoke-keys"
Sep 16 21:35:33.990235 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:33.990218    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffbf35ee-d22e-4aaf-b3bb-44995dc7397b" containerName="revoke-keys"
Sep 16 21:35:33.993317 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:33.993296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:33.997423 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:33.997393    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-35f5fd\"/\"simulator-w0-35f5fd-kserve-self-signed-certs\""
Sep 16 21:35:33.997533 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:33.997512    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-35f5fd\"/\"default-dockercfg-94lx6\""
Sep 16 21:35:33.997827 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:33.997807    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-35f5fd\"/\"kube-root-ca.crt\""
Sep 16 21:35:33.997907 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:33.997890    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-35f5fd\"/\"openshift-service-ca.crt\""
Sep 16 21:35:34.003563 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.003533    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"]
Sep 16 21:35:34.055215 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.055173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-home\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.055386 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.055246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwx46\" (UniqueName: \"kubernetes.io/projected/bd82ee24-7331-4ced-9aaa-80b6a3739348-kube-api-access-kwx46\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.055386 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.055341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-dshm\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.055386 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.055369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-model-cache\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.055557 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.055398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-tmp-dir\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.055557 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.055488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd82ee24-7331-4ced-9aaa-80b6a3739348-tls-certs\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.156151 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.156110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwx46\" (UniqueName: \"kubernetes.io/projected/bd82ee24-7331-4ced-9aaa-80b6a3739348-kube-api-access-kwx46\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.156348 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.156192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-dshm\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.156348 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.156217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-model-cache\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.156348 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.156245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-tmp-dir\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.156348 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.156287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd82ee24-7331-4ced-9aaa-80b6a3739348-tls-certs\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.156568 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.156362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-home\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.156732 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.156700    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-model-cache\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.156732 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.156729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-home\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.156934 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.156759    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-tmp-dir\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.158523 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.158504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-dshm\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.158943 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.158921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd82ee24-7331-4ced-9aaa-80b6a3739348-tls-certs\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.164352 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.164324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwx46\" (UniqueName: \"kubernetes.io/projected/bd82ee24-7331-4ced-9aaa-80b6a3739348-kube-api-access-kwx46\") pod \"simulator-w0-35f5fd-kserve-75559497f6-6tf4d\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") " pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.303275 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.303178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:34.572313 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:34.572285    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"]
Sep 16 21:35:34.573926 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:35:34.573894    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd82ee24_7331_4ced_9aaa_80b6a3739348.slice/crio-5b5c2edaec95e805e080c57066c807480135eed4090d2a61f263a49c46c0dd49 WatchSource:0}: Error finding container 5b5c2edaec95e805e080c57066c807480135eed4090d2a61f263a49c46c0dd49: Status 404 returned error can't find the container with id 5b5c2edaec95e805e080c57066c807480135eed4090d2a61f263a49c46c0dd49
Sep 16 21:35:35.322088 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:35.322048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d" event={"ID":"bd82ee24-7331-4ced-9aaa-80b6a3739348","Type":"ContainerStarted","Data":"335ffaee820105652f2f0a879804f9dc4831806589f014c328265508e4339f6e"}
Sep 16 21:35:35.322088 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:35.322095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d" event={"ID":"bd82ee24-7331-4ced-9aaa-80b6a3739348","Type":"ContainerStarted","Data":"5b5c2edaec95e805e080c57066c807480135eed4090d2a61f263a49c46c0dd49"}
Sep 16 21:35:35.341330 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:35.341276    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d" podStartSLOduration=2.341261465 podStartE2EDuration="2.341261465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:35:35.338440287 +0000 UTC m=+795.238650291" watchObservedRunningTime="2026-09-16 21:35:35.341261465 +0000 UTC m=+795.241471469"
Sep 16 21:35:36.549089 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:36.549055    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx"]
Sep 16 21:35:36.551591 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:36.551567    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-595d23dd-jr8vx"]
Sep 16 21:35:36.609729 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:36.609701    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffbf35ee-d22e-4aaf-b3bb-44995dc7397b" path="/var/lib/kubelet/pods/ffbf35ee-d22e-4aaf-b3bb-44995dc7397b/volumes"
Sep 16 21:35:36.682347 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:36.682317    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"]
Sep 16 21:35:36.682561 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:36.682525    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq" podUID="cd149156-3e03-4cf3-b77d-6e9da1fb11bc" containerName="istio-proxy" containerID="cri-o://93be242ff9fd39130032eb1c5149c8cfefc1055f995ca411dd13cd2c938d91cc" gracePeriod=30
Sep 16 21:35:37.414732 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.414695    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"]
Sep 16 21:35:37.418624 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.418603    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.420985 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.420964    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8eafdc51-openshift-default-dockercfg-6gv4j\""
Sep 16 21:35:37.434631 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.434609    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"]
Sep 16 21:35:37.498887 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.498857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istiod-ca-cert\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.499049 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.498891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-data\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.499049 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.498907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-podinfo\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.499049 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.498926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-workload-certs\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.499049 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.498996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4xw2\" (UniqueName: \"kubernetes.io/projected/28b394b8-8717-40b8-9a1c-84cbb3036b6f-kube-api-access-v4xw2\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.499049 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.499044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-token\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.499256 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.499122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-workload-socket\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.499256 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.499177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-envoy\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.499256 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.499228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-credential-socket\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.599954 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.599924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-envoy\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.600402 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.599978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-credential-socket\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.600402 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.600024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istiod-ca-cert\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.600402 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.600043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-data\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.600402 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.600059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-podinfo\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.600402 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.600075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-workload-certs\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.600402 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.600242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4xw2\" (UniqueName: \"kubernetes.io/projected/28b394b8-8717-40b8-9a1c-84cbb3036b6f-kube-api-access-v4xw2\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.600402 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.600292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-token\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.600402 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.600359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-workload-socket\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.600790 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.600431    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-workload-certs\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.600790 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.600478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-credential-socket\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.600862 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.600831    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istiod-ca-cert\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.600944 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.600914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-workload-socket\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.601140 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.601121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-data\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.602380 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.602356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-envoy\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.602493 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.602475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-podinfo\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.607844 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.607824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-token\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.608249 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.608231    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4xw2\" (UniqueName: \"kubernetes.io/projected/28b394b8-8717-40b8-9a1c-84cbb3036b6f-kube-api-access-v4xw2\") pod \"e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") " pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:37.729607 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:37.729574    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:38.011264 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:38.011224    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"]
Sep 16 21:35:38.012146 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:35:38.012119    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b394b8_8717_40b8_9a1c_84cbb3036b6f.slice/crio-e29b2bd604e1bf99737ac2a95d867465a0d68553ce66010239502d28b0952b75 WatchSource:0}: Error finding container e29b2bd604e1bf99737ac2a95d867465a0d68553ce66010239502d28b0952b75: Status 404 returned error can't find the container with id e29b2bd604e1bf99737ac2a95d867465a0d68553ce66010239502d28b0952b75
Sep 16 21:35:38.014814 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:38.014775    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:35:38.014884 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:38.014861    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:35:38.014930 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:38.014906    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:35:38.345242 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:38.343642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9" event={"ID":"28b394b8-8717-40b8-9a1c-84cbb3036b6f","Type":"ContainerStarted","Data":"924e2c5b50de50472faf129b6706b33478815b63e08f544e3d85e7164e1b0640"}
Sep 16 21:35:38.345242 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:38.343725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9" event={"ID":"28b394b8-8717-40b8-9a1c-84cbb3036b6f","Type":"ContainerStarted","Data":"e29b2bd604e1bf99737ac2a95d867465a0d68553ce66010239502d28b0952b75"}
Sep 16 21:35:38.364790 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:38.364748    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9" podStartSLOduration=1.364733966 podStartE2EDuration="1.364733966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:35:38.36295378 +0000 UTC m=+798.263163795" watchObservedRunningTime="2026-09-16 21:35:38.364733966 +0000 UTC m=+798.264943970"
Sep 16 21:35:38.730475 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:38.730448    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:39.735764 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:39.735734    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:40.353791 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:40.353760    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:40.354687 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:40.354648    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:35:41.947276 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:41.947209    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:42.047593 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.047560    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-workload-certs\") pod \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") "
Sep 16 21:35:42.047851 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.047623    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-data\") pod \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") "
Sep 16 21:35:42.047851 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.047680    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8h6x\" (UniqueName: \"kubernetes.io/projected/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-kube-api-access-g8h6x\") pod \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") "
Sep 16 21:35:42.047851 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.047800    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-podinfo\") pod \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") "
Sep 16 21:35:42.047851 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.047848    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-envoy\") pod \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") "
Sep 16 21:35:42.048071 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.047878    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istiod-ca-cert\") pod \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") "
Sep 16 21:35:42.048071 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.047936    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd149156-3e03-4cf3-b77d-6e9da1fb11bc" (UID: "cd149156-3e03-4cf3-b77d-6e9da1fb11bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:35:42.048071 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.047950    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-token\") pod \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") "
Sep 16 21:35:42.048071 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.047948    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd149156-3e03-4cf3-b77d-6e9da1fb11bc" (UID: "cd149156-3e03-4cf3-b77d-6e9da1fb11bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:35:42.048071 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.047997    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-credential-socket\") pod \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") "
Sep 16 21:35:42.048071 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.048037    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-workload-socket\") pod \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\" (UID: \"cd149156-3e03-4cf3-b77d-6e9da1fb11bc\") "
Sep 16 21:35:42.048373 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.048324    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd149156-3e03-4cf3-b77d-6e9da1fb11bc" (UID: "cd149156-3e03-4cf3-b77d-6e9da1fb11bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:35:42.048447 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.048422    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd149156-3e03-4cf3-b77d-6e9da1fb11bc" (UID: "cd149156-3e03-4cf3-b77d-6e9da1fb11bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:35:42.048513 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.048481    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:42.048513 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.048481    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd149156-3e03-4cf3-b77d-6e9da1fb11bc" (UID: "cd149156-3e03-4cf3-b77d-6e9da1fb11bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:35:42.048513 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.048494    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:42.048631 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.048516    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:42.050524 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.050502    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd149156-3e03-4cf3-b77d-6e9da1fb11bc" (UID: "cd149156-3e03-4cf3-b77d-6e9da1fb11bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:35:42.050595 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.050538    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd149156-3e03-4cf3-b77d-6e9da1fb11bc" (UID: "cd149156-3e03-4cf3-b77d-6e9da1fb11bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:35:42.050595 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.050555    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd149156-3e03-4cf3-b77d-6e9da1fb11bc" (UID: "cd149156-3e03-4cf3-b77d-6e9da1fb11bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:35:42.050595 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.050562    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-kube-api-access-g8h6x" (OuterVolumeSpecName: "kube-api-access-g8h6x") pod "cd149156-3e03-4cf3-b77d-6e9da1fb11bc" (UID: "cd149156-3e03-4cf3-b77d-6e9da1fb11bc"). InnerVolumeSpecName "kube-api-access-g8h6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:35:42.150002 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.149966    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8h6x\" (UniqueName: \"kubernetes.io/projected/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-kube-api-access-g8h6x\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:42.150002 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.149996    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:42.150002 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.150007    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:42.150231 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.150015    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:42.150231 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.150023    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:42.150231 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.150032    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd149156-3e03-4cf3-b77d-6e9da1fb11bc-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:35:42.366162 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.366076    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd149156-3e03-4cf3-b77d-6e9da1fb11bc" containerID="93be242ff9fd39130032eb1c5149c8cfefc1055f995ca411dd13cd2c938d91cc" exitCode=0
Sep 16 21:35:42.366333 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.366174    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"
Sep 16 21:35:42.366333 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.366212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq" event={"ID":"cd149156-3e03-4cf3-b77d-6e9da1fb11bc","Type":"ContainerDied","Data":"93be242ff9fd39130032eb1c5149c8cfefc1055f995ca411dd13cd2c938d91cc"}
Sep 16 21:35:42.366333 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.366299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq" event={"ID":"cd149156-3e03-4cf3-b77d-6e9da1fb11bc","Type":"ContainerDied","Data":"dfdc1df8f5eeaece2acf609b965560184b9d6c4d06d27fed0c1b9c128cc65ab5"}
Sep 16 21:35:42.366333 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.366323    2563 scope.go:117] "RemoveContainer" containerID="93be242ff9fd39130032eb1c5149c8cfefc1055f995ca411dd13cd2c938d91cc"
Sep 16 21:35:42.376631 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.376608    2563 scope.go:117] "RemoveContainer" containerID="eb6cc25649498ab4e2a3d0c84d7ec0455a10abea10a271b2ee9e2eb148d0c54b"
Sep 16 21:35:42.385580 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.385553    2563 scope.go:117] "RemoveContainer" containerID="93be242ff9fd39130032eb1c5149c8cfefc1055f995ca411dd13cd2c938d91cc"
Sep 16 21:35:42.386028 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:35:42.385991    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93be242ff9fd39130032eb1c5149c8cfefc1055f995ca411dd13cd2c938d91cc\": container with ID starting with 93be242ff9fd39130032eb1c5149c8cfefc1055f995ca411dd13cd2c938d91cc not found: ID does not exist" containerID="93be242ff9fd39130032eb1c5149c8cfefc1055f995ca411dd13cd2c938d91cc"
Sep 16 21:35:42.386158 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.386033    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93be242ff9fd39130032eb1c5149c8cfefc1055f995ca411dd13cd2c938d91cc"} err="failed to get container status \"93be242ff9fd39130032eb1c5149c8cfefc1055f995ca411dd13cd2c938d91cc\": rpc error: code = NotFound desc = could not find container \"93be242ff9fd39130032eb1c5149c8cfefc1055f995ca411dd13cd2c938d91cc\": container with ID starting with 93be242ff9fd39130032eb1c5149c8cfefc1055f995ca411dd13cd2c938d91cc not found: ID does not exist"
Sep 16 21:35:42.386158 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.386057    2563 scope.go:117] "RemoveContainer" containerID="eb6cc25649498ab4e2a3d0c84d7ec0455a10abea10a271b2ee9e2eb148d0c54b"
Sep 16 21:35:42.386403 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:35:42.386382    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6cc25649498ab4e2a3d0c84d7ec0455a10abea10a271b2ee9e2eb148d0c54b\": container with ID starting with eb6cc25649498ab4e2a3d0c84d7ec0455a10abea10a271b2ee9e2eb148d0c54b not found: ID does not exist" containerID="eb6cc25649498ab4e2a3d0c84d7ec0455a10abea10a271b2ee9e2eb148d0c54b"
Sep 16 21:35:42.386453 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.386411    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6cc25649498ab4e2a3d0c84d7ec0455a10abea10a271b2ee9e2eb148d0c54b"} err="failed to get container status \"eb6cc25649498ab4e2a3d0c84d7ec0455a10abea10a271b2ee9e2eb148d0c54b\": rpc error: code = NotFound desc = could not find container \"eb6cc25649498ab4e2a3d0c84d7ec0455a10abea10a271b2ee9e2eb148d0c54b\": container with ID starting with eb6cc25649498ab4e2a3d0c84d7ec0455a10abea10a271b2ee9e2eb148d0c54b not found: ID does not exist"
Sep 16 21:35:42.391510 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.391487    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"]
Sep 16 21:35:42.395683 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.395648    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-595d23dd-gw-openshift-default-f699c4b6d-bq7sq"]
Sep 16 21:35:42.612513 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:42.612468    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd149156-3e03-4cf3-b77d-6e9da1fb11bc" path="/var/lib/kubelet/pods/cd149156-3e03-4cf3-b77d-6e9da1fb11bc/volumes"
Sep 16 21:35:44.303889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:44.303803    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:44.303889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:44.303858    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:44.316449 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:44.316422    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:35:44.388711 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:35:44.388686    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:36:02.279052 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.279016    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"]
Sep 16 21:36:02.279495 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.279415    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd149156-3e03-4cf3-b77d-6e9da1fb11bc" containerName="istio-proxy"
Sep 16 21:36:02.279495 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.279427    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd149156-3e03-4cf3-b77d-6e9da1fb11bc" containerName="istio-proxy"
Sep 16 21:36:02.279577 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.279508    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd149156-3e03-4cf3-b77d-6e9da1fb11bc" containerName="istio-proxy"
Sep 16 21:36:02.282940 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.282920    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.285182 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.285157    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e5568d\"/\"openshift-service-ca.crt\""
Sep 16 21:36:02.285389 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.285370    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e5568d\"/\"kube-root-ca.crt\""
Sep 16 21:36:02.285995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.285950    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e5568d\"/\"test-model-e5568d-kserve-self-signed-certs\""
Sep 16 21:36:02.285995 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.285984    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e5568d\"/\"default-dockercfg-mb5pm\""
Sep 16 21:36:02.292026 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.291372    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"]
Sep 16 21:36:02.368440 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.368397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psltz\" (UniqueName: \"kubernetes.io/projected/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-kube-api-access-psltz\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.368627 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.368458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-dshm\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.368627 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.368518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-tls-certs\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.368627 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.368597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-model-cache\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.368627 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.368625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-tmp-dir\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.368871 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.368779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-home\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.470033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.470001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-home\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.470221 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.470051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psltz\" (UniqueName: \"kubernetes.io/projected/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-kube-api-access-psltz\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.470221 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.470082    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-dshm\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.470221 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.470107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-tls-certs\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.470221 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.470191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-model-cache\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.470438 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.470224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-tmp-dir\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.470531 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.470508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-home\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.470774 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.470752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-model-cache\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.470858 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.470793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-tmp-dir\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.472700 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.472644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-dshm\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.473220 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.473195    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-tls-certs\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.478156 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.478134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psltz\" (UniqueName: \"kubernetes.io/projected/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-kube-api-access-psltz\") pod \"test-model-e5568d-kserve-7c5884779f-6bxjq\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") " pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.594830 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.594736    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:02.867694 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:02.867650    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"]
Sep 16 21:36:02.869413 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:36:02.869379    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0905dbe8_baa8_4af3_9ef2_4d4f058f6c12.slice/crio-fb07be037e1f2ad5dc824df85af81c279865eba3d60077e0cf75be9dfb339065 WatchSource:0}: Error finding container fb07be037e1f2ad5dc824df85af81c279865eba3d60077e0cf75be9dfb339065: Status 404 returned error can't find the container with id fb07be037e1f2ad5dc824df85af81c279865eba3d60077e0cf75be9dfb339065
Sep 16 21:36:03.488806 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:03.488711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq" event={"ID":"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12","Type":"ContainerStarted","Data":"9dfc2779973a49e3b28b6c24529c1325aba8c26d9fa0a1c36c27f8ae196c5910"}
Sep 16 21:36:03.488806 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:03.488771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq" event={"ID":"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12","Type":"ContainerStarted","Data":"fb07be037e1f2ad5dc824df85af81c279865eba3d60077e0cf75be9dfb339065"}
Sep 16 21:36:03.492562 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:03.492520    2563 generic.go:356] "Generic (PLEG): container finished" podID="28b394b8-8717-40b8-9a1c-84cbb3036b6f" containerID="924e2c5b50de50472faf129b6706b33478815b63e08f544e3d85e7164e1b0640" exitCode=0
Sep 16 21:36:03.492737 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:03.492588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9" event={"ID":"28b394b8-8717-40b8-9a1c-84cbb3036b6f","Type":"ContainerDied","Data":"924e2c5b50de50472faf129b6706b33478815b63e08f544e3d85e7164e1b0640"}
Sep 16 21:36:03.493061 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:03.493039    2563 scope.go:117] "RemoveContainer" containerID="924e2c5b50de50472faf129b6706b33478815b63e08f544e3d85e7164e1b0640"
Sep 16 21:36:03.494483 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:03.494449    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:36:03.494590 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:03.494534    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:36:03.494679 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:03.494586    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:36:03.509512 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:03.509466    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq" podStartSLOduration=1.509450596 podStartE2EDuration="1.509450596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:36:03.506594243 +0000 UTC m=+823.406804248" watchObservedRunningTime="2026-09-16 21:36:03.509450596 +0000 UTC m=+823.409660599"
Sep 16 21:36:04.500224 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:04.500189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9" event={"ID":"28b394b8-8717-40b8-9a1c-84cbb3036b6f","Type":"ContainerStarted","Data":"df9144fc23dd8ae5f8a327417d278621b868b6680ea12603c787915177c099ac"}
Sep 16 21:36:04.730705 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:04.730648    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:36:04.730916 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:04.730893    2563 patch_prober.go:28] interesting pod/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 16 21:36:04.730964 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:04.730948    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9" podUID="28b394b8-8717-40b8-9a1c-84cbb3036b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 16 21:36:05.428060 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:05.428028    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2"]
Sep 16 21:36:05.428528 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:05.428510    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd149156-3e03-4cf3-b77d-6e9da1fb11bc" containerName="istio-proxy"
Sep 16 21:36:05.428627 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:05.428530    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd149156-3e03-4cf3-b77d-6e9da1fb11bc" containerName="istio-proxy"
Sep 16 21:36:05.428706 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:05.428682    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd149156-3e03-4cf3-b77d-6e9da1fb11bc" containerName="istio-proxy"
Sep 16 21:36:05.431921 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:05.431899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2"
Sep 16 21:36:05.434140 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:05.434116    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:36:05.434540 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:05.434522    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:36:05.440403 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:05.440378    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2"]
Sep 16 21:36:05.506023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:05.505988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe70879-5d10-41a8-be0e-513a2954560b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2\" (UID: \"5fe70879-5d10-41a8-be0e-513a2954560b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2"
Sep 16 21:36:05.607374 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:05.607332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe70879-5d10-41a8-be0e-513a2954560b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2\" (UID: \"5fe70879-5d10-41a8-be0e-513a2954560b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2"
Sep 16 21:36:05.608020 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:05.607999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe70879-5d10-41a8-be0e-513a2954560b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2\" (UID: \"5fe70879-5d10-41a8-be0e-513a2954560b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2"
Sep 16 21:36:05.731069 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:05.730984    2563 patch_prober.go:28] interesting pod/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 16 21:36:05.731213 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:05.731068    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9" podUID="28b394b8-8717-40b8-9a1c-84cbb3036b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 16 21:36:05.744022 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:05.743996    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2"
Sep 16 21:36:06.067963 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:06.065776    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2"]
Sep 16 21:36:06.075902 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:36:06.075787    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe70879_5d10_41a8_be0e_513a2954560b.slice/crio-f647bf3f237a093c507c0b558a874df294dfe34ae8549c4b2f6c62297587a525 WatchSource:0}: Error finding container f647bf3f237a093c507c0b558a874df294dfe34ae8549c4b2f6c62297587a525: Status 404 returned error can't find the container with id f647bf3f237a093c507c0b558a874df294dfe34ae8549c4b2f6c62297587a525
Sep 16 21:36:06.513219 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:06.513185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2" event={"ID":"5fe70879-5d10-41a8-be0e-513a2954560b","Type":"ContainerStarted","Data":"f42e3c6663d923270d8032e9000c81bc386f5d6c4c95aa3e9610b0de513770bc"}
Sep 16 21:36:06.513614 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:06.513235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2" event={"ID":"5fe70879-5d10-41a8-be0e-513a2954560b","Type":"ContainerStarted","Data":"f647bf3f237a093c507c0b558a874df294dfe34ae8549c4b2f6c62297587a525"}
Sep 16 21:36:06.531491 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:06.531440    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2" podStartSLOduration=1.531424586 podStartE2EDuration="1.531424586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:36:06.530477202 +0000 UTC m=+826.430687206" watchObservedRunningTime="2026-09-16 21:36:06.531424586 +0000 UTC m=+826.431634588"
Sep 16 21:36:06.730392 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:06.730310    2563 patch_prober.go:28] interesting pod/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 16 21:36:06.730392 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:06.730380    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9" podUID="28b394b8-8717-40b8-9a1c-84cbb3036b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 16 21:36:07.519529 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:07.519495    2563 generic.go:356] "Generic (PLEG): container finished" podID="5fe70879-5d10-41a8-be0e-513a2954560b" containerID="f42e3c6663d923270d8032e9000c81bc386f5d6c4c95aa3e9610b0de513770bc" exitCode=0
Sep 16 21:36:07.519940 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:07.519631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2" event={"ID":"5fe70879-5d10-41a8-be0e-513a2954560b","Type":"ContainerDied","Data":"f42e3c6663d923270d8032e9000c81bc386f5d6c4c95aa3e9610b0de513770bc"}
Sep 16 21:36:07.730638 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:07.730602    2563 patch_prober.go:28] interesting pod/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 16 21:36:07.730837 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:07.730683    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9" podUID="28b394b8-8717-40b8-9a1c-84cbb3036b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 16 21:36:08.720577 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:08.720549    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2"
Sep 16 21:36:08.730675 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:08.730589    2563 patch_prober.go:28] interesting pod/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 16 21:36:08.730832 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:08.730673    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9" podUID="28b394b8-8717-40b8-9a1c-84cbb3036b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 16 21:36:08.838742 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:08.838625    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe70879-5d10-41a8-be0e-513a2954560b-maas-api-service-ca\") pod \"5fe70879-5d10-41a8-be0e-513a2954560b\" (UID: \"5fe70879-5d10-41a8-be0e-513a2954560b\") "
Sep 16 21:36:08.838988 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:08.838965    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe70879-5d10-41a8-be0e-513a2954560b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5fe70879-5d10-41a8-be0e-513a2954560b" (UID: "5fe70879-5d10-41a8-be0e-513a2954560b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:36:08.939694 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:08.939648    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe70879-5d10-41a8-be0e-513a2954560b-maas-api-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:09.533802 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:09.533772    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2"
Sep 16 21:36:09.533982 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:09.533769    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2" event={"ID":"5fe70879-5d10-41a8-be0e-513a2954560b","Type":"ContainerDied","Data":"f647bf3f237a093c507c0b558a874df294dfe34ae8549c4b2f6c62297587a525"}
Sep 16 21:36:09.533982 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:09.533893    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f647bf3f237a093c507c0b558a874df294dfe34ae8549c4b2f6c62297587a525"
Sep 16 21:36:09.734748 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:09.734719    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:36:09.735181 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:09.734987    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:36:09.735773 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:09.735755    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:36:12.193146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:12.193113    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2"]
Sep 16 21:36:12.195646 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:12.195623    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8eafdc51-th6v2"]
Sep 16 21:36:12.595430 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:12.595346    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:12.595430 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:12.595396    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:12.608354 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:12.608326    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fe70879-5d10-41a8-be0e-513a2954560b" path="/var/lib/kubelet/pods/5fe70879-5d10-41a8-be0e-513a2954560b/volumes"
Sep 16 21:36:12.608630 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:12.608618    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:13.320387 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:13.320358    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"]
Sep 16 21:36:13.320848 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:13.320569    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9" podUID="28b394b8-8717-40b8-9a1c-84cbb3036b6f" containerName="istio-proxy" containerID="cri-o://df9144fc23dd8ae5f8a327417d278621b868b6680ea12603c787915177c099ac" gracePeriod=30
Sep 16 21:36:13.571170 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:13.571083    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:36:14.145848 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.145817    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"]
Sep 16 21:36:14.146188 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.146176    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe70879-5d10-41a8-be0e-513a2954560b" containerName="revoke-keys"
Sep 16 21:36:14.146248 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.146191    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe70879-5d10-41a8-be0e-513a2954560b" containerName="revoke-keys"
Sep 16 21:36:14.146316 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.146305    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe70879-5d10-41a8-be0e-513a2954560b" containerName="revoke-keys"
Sep 16 21:36:14.149463 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.149447    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.153048 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.153026    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a1fb6d89-openshift-default-dockercfg-prx86\""
Sep 16 21:36:14.181240 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.181212    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"]
Sep 16 21:36:14.297906 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.297860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istiod-ca-cert\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.298116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.297923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg5jd\" (UniqueName: \"kubernetes.io/projected/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-kube-api-access-rg5jd\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.298116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.297994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-token\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.298116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.298066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-data\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.298116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.298098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-envoy\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.298340 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.298137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-credential-socket\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.298340 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.298176    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-workload-certs\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.298340 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.298231    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-workload-socket\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.298340 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.298265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-podinfo\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.399181 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.399097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istiod-ca-cert\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.399181 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.399145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg5jd\" (UniqueName: \"kubernetes.io/projected/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-kube-api-access-rg5jd\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.399758 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.399201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-token\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.399758 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.399263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-data\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.399758 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.399291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-envoy\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.399758 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.399327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-credential-socket\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.399758 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.399366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-workload-certs\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.399758 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.399414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-workload-socket\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.400074 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.399528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-podinfo\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.400074 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.399907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istiod-ca-cert\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.400074 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.399699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-data\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.400074 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.399825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-workload-socket\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.400074 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.399790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-workload-certs\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.400074 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.399890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-credential-socket\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.401804 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.401786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-envoy\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.402032 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.402014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-podinfo\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.419558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.419530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg5jd\" (UniqueName: \"kubernetes.io/projected/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-kube-api-access-rg5jd\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.438416 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.438389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-token\") pod \"e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") " pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.459741 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.459719    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:14.755047 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.755020    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"]
Sep 16 21:36:14.757159 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:36:14.757130    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c75e61_24ae_4d13_8a3c_ee68158b1f2d.slice/crio-213cd136230db4989944a4a37483f0cc413952c5c3aacce8a469b4fbc4ae0a88 WatchSource:0}: Error finding container 213cd136230db4989944a4a37483f0cc413952c5c3aacce8a469b4fbc4ae0a88: Status 404 returned error can't find the container with id 213cd136230db4989944a4a37483f0cc413952c5c3aacce8a469b4fbc4ae0a88
Sep 16 21:36:14.759521 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.759487    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:36:14.759590 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.759555    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:36:14.759590 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:14.759582    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:36:15.569410 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:15.569313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78" event={"ID":"60c75e61-24ae-4d13-8a3c-ee68158b1f2d","Type":"ContainerStarted","Data":"a0bf9670658bf5115e5e6b4c309c3f034cdda5f00c1a51312d5394fed095d289"}
Sep 16 21:36:15.569410 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:15.569360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78" event={"ID":"60c75e61-24ae-4d13-8a3c-ee68158b1f2d","Type":"ContainerStarted","Data":"213cd136230db4989944a4a37483f0cc413952c5c3aacce8a469b4fbc4ae0a88"}
Sep 16 21:36:15.590179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:15.590127    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78" podStartSLOduration=1.590113151 podStartE2EDuration="1.590113151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:36:15.588791825 +0000 UTC m=+835.489001832" watchObservedRunningTime="2026-09-16 21:36:15.590113151 +0000 UTC m=+835.490323155"
Sep 16 21:36:16.460025 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:16.459987    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:16.465554 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:16.465527    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:16.575369 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:16.575335    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:16.576340 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:16.576320    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:18.575413 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.575393    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:36:18.588872 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.588786    2563 generic.go:356] "Generic (PLEG): container finished" podID="28b394b8-8717-40b8-9a1c-84cbb3036b6f" containerID="df9144fc23dd8ae5f8a327417d278621b868b6680ea12603c787915177c099ac" exitCode=0
Sep 16 21:36:18.588872 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.588843    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"
Sep 16 21:36:18.589081 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.588873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9" event={"ID":"28b394b8-8717-40b8-9a1c-84cbb3036b6f","Type":"ContainerDied","Data":"df9144fc23dd8ae5f8a327417d278621b868b6680ea12603c787915177c099ac"}
Sep 16 21:36:18.589081 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.588924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9" event={"ID":"28b394b8-8717-40b8-9a1c-84cbb3036b6f","Type":"ContainerDied","Data":"e29b2bd604e1bf99737ac2a95d867465a0d68553ce66010239502d28b0952b75"}
Sep 16 21:36:18.589081 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.588951    2563 scope.go:117] "RemoveContainer" containerID="df9144fc23dd8ae5f8a327417d278621b868b6680ea12603c787915177c099ac"
Sep 16 21:36:18.598827 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.598805    2563 scope.go:117] "RemoveContainer" containerID="924e2c5b50de50472faf129b6706b33478815b63e08f544e3d85e7164e1b0640"
Sep 16 21:36:18.610475 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.610435    2563 scope.go:117] "RemoveContainer" containerID="df9144fc23dd8ae5f8a327417d278621b868b6680ea12603c787915177c099ac"
Sep 16 21:36:18.613414 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:36:18.613368    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df9144fc23dd8ae5f8a327417d278621b868b6680ea12603c787915177c099ac\": container with ID starting with df9144fc23dd8ae5f8a327417d278621b868b6680ea12603c787915177c099ac not found: ID does not exist" containerID="df9144fc23dd8ae5f8a327417d278621b868b6680ea12603c787915177c099ac"
Sep 16 21:36:18.613538 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.613419    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df9144fc23dd8ae5f8a327417d278621b868b6680ea12603c787915177c099ac"} err="failed to get container status \"df9144fc23dd8ae5f8a327417d278621b868b6680ea12603c787915177c099ac\": rpc error: code = NotFound desc = could not find container \"df9144fc23dd8ae5f8a327417d278621b868b6680ea12603c787915177c099ac\": container with ID starting with df9144fc23dd8ae5f8a327417d278621b868b6680ea12603c787915177c099ac not found: ID does not exist"
Sep 16 21:36:18.613538 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.613444    2563 scope.go:117] "RemoveContainer" containerID="924e2c5b50de50472faf129b6706b33478815b63e08f544e3d85e7164e1b0640"
Sep 16 21:36:18.613910 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:36:18.613881    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"924e2c5b50de50472faf129b6706b33478815b63e08f544e3d85e7164e1b0640\": container with ID starting with 924e2c5b50de50472faf129b6706b33478815b63e08f544e3d85e7164e1b0640 not found: ID does not exist" containerID="924e2c5b50de50472faf129b6706b33478815b63e08f544e3d85e7164e1b0640"
Sep 16 21:36:18.614003 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.613920    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"924e2c5b50de50472faf129b6706b33478815b63e08f544e3d85e7164e1b0640"} err="failed to get container status \"924e2c5b50de50472faf129b6706b33478815b63e08f544e3d85e7164e1b0640\": rpc error: code = NotFound desc = could not find container \"924e2c5b50de50472faf129b6706b33478815b63e08f544e3d85e7164e1b0640\": container with ID starting with 924e2c5b50de50472faf129b6706b33478815b63e08f544e3d85e7164e1b0640 not found: ID does not exist"
Sep 16 21:36:18.644403 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.644366    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-envoy\") pod \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") "
Sep 16 21:36:18.644574 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.644479    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4xw2\" (UniqueName: \"kubernetes.io/projected/28b394b8-8717-40b8-9a1c-84cbb3036b6f-kube-api-access-v4xw2\") pod \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") "
Sep 16 21:36:18.644574 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.644523    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-credential-socket\") pod \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") "
Sep 16 21:36:18.644710 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.644590    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istiod-ca-cert\") pod \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") "
Sep 16 21:36:18.644710 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.644614    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-workload-certs\") pod \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") "
Sep 16 21:36:18.644830 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.644737    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-workload-socket\") pod \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") "
Sep 16 21:36:18.644830 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.644789    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-data\") pod \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") "
Sep 16 21:36:18.644830 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.644812    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-podinfo\") pod \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") "
Sep 16 21:36:18.645001 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.644892    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-token\") pod \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\" (UID: \"28b394b8-8717-40b8-9a1c-84cbb3036b6f\") "
Sep 16 21:36:18.645001 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.644946    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "28b394b8-8717-40b8-9a1c-84cbb3036b6f" (UID: "28b394b8-8717-40b8-9a1c-84cbb3036b6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:18.645101 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.645047    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "28b394b8-8717-40b8-9a1c-84cbb3036b6f" (UID: "28b394b8-8717-40b8-9a1c-84cbb3036b6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:36:18.645240 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.645216    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "28b394b8-8717-40b8-9a1c-84cbb3036b6f" (UID: "28b394b8-8717-40b8-9a1c-84cbb3036b6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:18.645464 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.645431    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-data" (OuterVolumeSpecName: "istio-data") pod "28b394b8-8717-40b8-9a1c-84cbb3036b6f" (UID: "28b394b8-8717-40b8-9a1c-84cbb3036b6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:18.645551 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.645445    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "28b394b8-8717-40b8-9a1c-84cbb3036b6f" (UID: "28b394b8-8717-40b8-9a1c-84cbb3036b6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:18.645612 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.645562    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:18.645612 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.645583    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:18.645612 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.645597    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:18.645786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.645610    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:18.647445 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.647402    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b394b8-8717-40b8-9a1c-84cbb3036b6f-kube-api-access-v4xw2" (OuterVolumeSpecName: "kube-api-access-v4xw2") pod "28b394b8-8717-40b8-9a1c-84cbb3036b6f" (UID: "28b394b8-8717-40b8-9a1c-84cbb3036b6f"). InnerVolumeSpecName "kube-api-access-v4xw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:18.647726 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.647701    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-token" (OuterVolumeSpecName: "istio-token") pod "28b394b8-8717-40b8-9a1c-84cbb3036b6f" (UID: "28b394b8-8717-40b8-9a1c-84cbb3036b6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:18.648047 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.647993    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "28b394b8-8717-40b8-9a1c-84cbb3036b6f" (UID: "28b394b8-8717-40b8-9a1c-84cbb3036b6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:36:18.648189 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.648163    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "28b394b8-8717-40b8-9a1c-84cbb3036b6f" (UID: "28b394b8-8717-40b8-9a1c-84cbb3036b6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:18.746616 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.746583    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:18.746616 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.746617    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:18.746819 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.746634    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:18.746819 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.746649    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4xw2\" (UniqueName: \"kubernetes.io/projected/28b394b8-8717-40b8-9a1c-84cbb3036b6f-kube-api-access-v4xw2\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:18.746819 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.746690    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b394b8-8717-40b8-9a1c-84cbb3036b6f-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:18.920639 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.920607    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"]
Sep 16 21:36:18.932590 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:18.932562    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8eafdc51-openshift-default-cfc745794-7hxd9"]
Sep 16 21:36:20.109098 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:20.109061    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd"]
Sep 16 21:36:20.109622 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:20.109602    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28b394b8-8717-40b8-9a1c-84cbb3036b6f" containerName="istio-proxy"
Sep 16 21:36:20.109752 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:20.109625    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="28b394b8-8717-40b8-9a1c-84cbb3036b6f" containerName="istio-proxy"
Sep 16 21:36:20.109808 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:20.109770    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="28b394b8-8717-40b8-9a1c-84cbb3036b6f" containerName="istio-proxy"
Sep 16 21:36:20.109808 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:20.109788    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="28b394b8-8717-40b8-9a1c-84cbb3036b6f" containerName="istio-proxy"
Sep 16 21:36:20.114088 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:20.114067    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd"
Sep 16 21:36:20.117743 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:20.117722    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:36:20.118015 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:20.117995    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:36:20.122969 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:20.122944    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd"]
Sep 16 21:36:20.262462 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:20.262423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfc7f5e-228a-44a9-9452-193995673ae8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd\" (UID: \"abfc7f5e-228a-44a9-9452-193995673ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd"
Sep 16 21:36:20.363627 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:20.363520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfc7f5e-228a-44a9-9452-193995673ae8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd\" (UID: \"abfc7f5e-228a-44a9-9452-193995673ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd"
Sep 16 21:36:20.364191 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:20.364171    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfc7f5e-228a-44a9-9452-193995673ae8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd\" (UID: \"abfc7f5e-228a-44a9-9452-193995673ae8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd"
Sep 16 21:36:20.425410 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:20.425377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd"
Sep 16 21:36:20.609829 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:20.609780    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28b394b8-8717-40b8-9a1c-84cbb3036b6f" path="/var/lib/kubelet/pods/28b394b8-8717-40b8-9a1c-84cbb3036b6f/volumes"
Sep 16 21:36:20.693227 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:20.693207    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd"]
Sep 16 21:36:20.694929 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:36:20.694892    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfc7f5e_228a_44a9_9452_193995673ae8.slice/crio-63bc419bc096038aeb2048f3d913aa3d67b954909f71e46c21db3c91c56f8d6b WatchSource:0}: Error finding container 63bc419bc096038aeb2048f3d913aa3d67b954909f71e46c21db3c91c56f8d6b: Status 404 returned error can't find the container with id 63bc419bc096038aeb2048f3d913aa3d67b954909f71e46c21db3c91c56f8d6b
Sep 16 21:36:21.606992 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:21.606945    2563 generic.go:356] "Generic (PLEG): container finished" podID="abfc7f5e-228a-44a9-9452-193995673ae8" containerID="813d57a1b98faae2565c58dc89bbbacffb547d05d48b01729ab9d456a27d5884" exitCode=0
Sep 16 21:36:21.606992 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:21.606995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd" event={"ID":"abfc7f5e-228a-44a9-9452-193995673ae8","Type":"ContainerDied","Data":"813d57a1b98faae2565c58dc89bbbacffb547d05d48b01729ab9d456a27d5884"}
Sep 16 21:36:21.607712 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:21.607076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd" event={"ID":"abfc7f5e-228a-44a9-9452-193995673ae8","Type":"ContainerStarted","Data":"63bc419bc096038aeb2048f3d913aa3d67b954909f71e46c21db3c91c56f8d6b"}
Sep 16 21:36:22.740056 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:22.740032    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd"
Sep 16 21:36:22.888793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:22.888714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfc7f5e-228a-44a9-9452-193995673ae8-maas-api-service-ca\") pod \"abfc7f5e-228a-44a9-9452-193995673ae8\" (UID: \"abfc7f5e-228a-44a9-9452-193995673ae8\") "
Sep 16 21:36:22.889138 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:22.889111    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abfc7f5e-228a-44a9-9452-193995673ae8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abfc7f5e-228a-44a9-9452-193995673ae8" (UID: "abfc7f5e-228a-44a9-9452-193995673ae8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:36:22.990043 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:22.990011    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abfc7f5e-228a-44a9-9452-193995673ae8-maas-api-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:23.618673 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:23.618626    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd"
Sep 16 21:36:23.618673 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:23.618618    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd" event={"ID":"abfc7f5e-228a-44a9-9452-193995673ae8","Type":"ContainerDied","Data":"63bc419bc096038aeb2048f3d913aa3d67b954909f71e46c21db3c91c56f8d6b"}
Sep 16 21:36:23.618878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:23.618710    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63bc419bc096038aeb2048f3d913aa3d67b954909f71e46c21db3c91c56f8d6b"
Sep 16 21:36:31.177345 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:31.177311    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd"]
Sep 16 21:36:31.180218 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:31.180187    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-ddbefa-xn7wd"]
Sep 16 21:36:32.609650 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:32.609620    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abfc7f5e-228a-44a9-9452-193995673ae8" path="/var/lib/kubelet/pods/abfc7f5e-228a-44a9-9452-193995673ae8/volumes"
Sep 16 21:36:43.175323 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:43.175280    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"]
Sep 16 21:36:43.175830 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:43.175563    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x" podUID="ebb4c011-6706-4815-83a0-8bccd981be05" containerName="istio-proxy" containerID="cri-o://cc11bddc036aa43df220623d56ad5744bbdc77eb65118ff77224df5dfe86a9d5" gracePeriod=30
Sep 16 21:36:43.934546 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:43.934515    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"]
Sep 16 21:36:43.934962 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:43.934948    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abfc7f5e-228a-44a9-9452-193995673ae8" containerName="revoke-keys"
Sep 16 21:36:43.935016 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:43.934964    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="abfc7f5e-228a-44a9-9452-193995673ae8" containerName="revoke-keys"
Sep 16 21:36:43.935016 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:43.934989    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28b394b8-8717-40b8-9a1c-84cbb3036b6f" containerName="istio-proxy"
Sep 16 21:36:43.935016 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:43.934995    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="28b394b8-8717-40b8-9a1c-84cbb3036b6f" containerName="istio-proxy"
Sep 16 21:36:43.935115 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:43.935061    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="abfc7f5e-228a-44a9-9452-193995673ae8" containerName="revoke-keys"
Sep 16 21:36:43.938117 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:43.938097    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:43.941083 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:43.941059    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-b91829-openshift-default-dockercfg-zr582\""
Sep 16 21:36:43.953605 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:43.953581    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"]
Sep 16 21:36:44.029763 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.029667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gzml\" (UniqueName: \"kubernetes.io/projected/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-kube-api-access-8gzml\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.029991 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.029970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-workload-socket\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.030264 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.030245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-podinfo\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.030422 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.030405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-workload-certs\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.030523 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.030510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-token\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.030643 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.030630    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-credential-socket\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.030785 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.030769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-data\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.030891 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.030877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istiod-ca-cert\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.030996 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.030982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-envoy\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.132608 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.132575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-data\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.132795 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.132623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istiod-ca-cert\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.132795 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.132688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-envoy\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.132795 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.132782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gzml\" (UniqueName: \"kubernetes.io/projected/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-kube-api-access-8gzml\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.132938 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.132811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-workload-socket\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.132938 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.132914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-podinfo\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.133048 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.132974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-workload-certs\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.133048 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.133000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-token\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.133048 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.133007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-data\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.133048 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.133041    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-credential-socket\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.133260 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.133214    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-workload-socket\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.133319 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.133265    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-workload-certs\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.133396 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.133372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-credential-socket\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.133502 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.133480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istiod-ca-cert\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.135070 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.135047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-envoy\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.135537 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.135516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-podinfo\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.140479 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.140456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gzml\" (UniqueName: \"kubernetes.io/projected/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-kube-api-access-8gzml\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.140563 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.140458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-token\") pod \"e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") " pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.248455 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.248416    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:44.527845 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.527809    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"]
Sep 16 21:36:44.529840 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:36:44.529809    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8576f11f_bd1e_4c61_8a0f_fdac1fe4467a.slice/crio-4704ba08e1d81862682453bc1989e472948afd46e87206d6d30ae7fc5bb4d62e WatchSource:0}: Error finding container 4704ba08e1d81862682453bc1989e472948afd46e87206d6d30ae7fc5bb4d62e: Status 404 returned error can't find the container with id 4704ba08e1d81862682453bc1989e472948afd46e87206d6d30ae7fc5bb4d62e
Sep 16 21:36:44.532131 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.532096    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:36:44.532247 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.532171    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:36:44.532247 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.532200    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:36:44.727071 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.727034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82" event={"ID":"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a","Type":"ContainerStarted","Data":"34110ba7cc805bcd8f5429e8cc6232bf95c4bc2997bffda85ccf24f54b311950"}
Sep 16 21:36:44.727281 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.727096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82" event={"ID":"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a","Type":"ContainerStarted","Data":"4704ba08e1d81862682453bc1989e472948afd46e87206d6d30ae7fc5bb4d62e"}
Sep 16 21:36:44.747739 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:44.747687    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82" podStartSLOduration=1.7476703630000001 podStartE2EDuration="1.747670363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:36:44.746693114 +0000 UTC m=+864.646903119" watchObservedRunningTime="2026-09-16 21:36:44.747670363 +0000 UTC m=+864.647880380"
Sep 16 21:36:45.249173 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:45.249136    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:45.254495 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:45.254472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:45.732496 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:45.732464    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:45.733590 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:45.733569    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:36:46.623058 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:46.621239    2563 patch_prober.go:28] interesting pod/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": read tcp 10.133.0.2:34630->10.133.0.47:15021: read: connection reset by peer" start-of-body=
Sep 16 21:36:46.623058 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:46.621319    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78" podUID="60c75e61-24ae-4d13-8a3c-ee68158b1f2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": read tcp 10.133.0.2:34630->10.133.0.47:15021: read: connection reset by peer"
Sep 16 21:36:46.739604 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:46.739570    2563 generic.go:356] "Generic (PLEG): container finished" podID="60c75e61-24ae-4d13-8a3c-ee68158b1f2d" containerID="a0bf9670658bf5115e5e6b4c309c3f034cdda5f00c1a51312d5394fed095d289" exitCode=0
Sep 16 21:36:46.739821 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:46.739646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78" event={"ID":"60c75e61-24ae-4d13-8a3c-ee68158b1f2d","Type":"ContainerDied","Data":"a0bf9670658bf5115e5e6b4c309c3f034cdda5f00c1a51312d5394fed095d289"}
Sep 16 21:36:46.740226 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:46.740195    2563 scope.go:117] "RemoveContainer" containerID="a0bf9670658bf5115e5e6b4c309c3f034cdda5f00c1a51312d5394fed095d289"
Sep 16 21:36:46.741270 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:46.741243    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:36:46.741336 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:46.741311    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:36:46.741375 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:46.741344    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:36:47.746903 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:47.746856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78" event={"ID":"60c75e61-24ae-4d13-8a3c-ee68158b1f2d","Type":"ContainerStarted","Data":"89829aef533fe975f94c958eed2ef6db884a6dea7b069c8cdcfd08ab074568a6"}
Sep 16 21:36:48.445797 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.445773    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:36:48.460104 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.460035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:48.460313 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.460295    2563 patch_prober.go:28] interesting pod/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 16 21:36:48.460363 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.460338    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78" podUID="60c75e61-24ae-4d13-8a3c-ee68158b1f2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 16 21:36:48.588889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.587040    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb4c011-6706-4815-83a0-8bccd981be05-istiod-ca-cert\") pod \"ebb4c011-6706-4815-83a0-8bccd981be05\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") "
Sep 16 21:36:48.588889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.587108    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb4c011-6706-4815-83a0-8bccd981be05-istio-podinfo\") pod \"ebb4c011-6706-4815-83a0-8bccd981be05\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") "
Sep 16 21:36:48.588889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.587156    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-credential-socket\") pod \"ebb4c011-6706-4815-83a0-8bccd981be05\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") "
Sep 16 21:36:48.588889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.587189    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-istio-data\") pod \"ebb4c011-6706-4815-83a0-8bccd981be05\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") "
Sep 16 21:36:48.588889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.587249    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb4c011-6706-4815-83a0-8bccd981be05-istio-token\") pod \"ebb4c011-6706-4815-83a0-8bccd981be05\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") "
Sep 16 21:36:48.588889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.587328    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-workload-socket\") pod \"ebb4c011-6706-4815-83a0-8bccd981be05\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") "
Sep 16 21:36:48.588889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.587357    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-istio-envoy\") pod \"ebb4c011-6706-4815-83a0-8bccd981be05\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") "
Sep 16 21:36:48.588889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.587386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-workload-certs\") pod \"ebb4c011-6706-4815-83a0-8bccd981be05\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") "
Sep 16 21:36:48.588889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.587417    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76jfs\" (UniqueName: \"kubernetes.io/projected/ebb4c011-6706-4815-83a0-8bccd981be05-kube-api-access-76jfs\") pod \"ebb4c011-6706-4815-83a0-8bccd981be05\" (UID: \"ebb4c011-6706-4815-83a0-8bccd981be05\") "
Sep 16 21:36:48.588889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.588346    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-istio-data" (OuterVolumeSpecName: "istio-data") pod "ebb4c011-6706-4815-83a0-8bccd981be05" (UID: "ebb4c011-6706-4815-83a0-8bccd981be05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:48.588889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.588608    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb4c011-6706-4815-83a0-8bccd981be05-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ebb4c011-6706-4815-83a0-8bccd981be05" (UID: "ebb4c011-6706-4815-83a0-8bccd981be05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:36:48.589927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.589897    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ebb4c011-6706-4815-83a0-8bccd981be05" (UID: "ebb4c011-6706-4815-83a0-8bccd981be05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:48.590124 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.589976    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ebb4c011-6706-4815-83a0-8bccd981be05" (UID: "ebb4c011-6706-4815-83a0-8bccd981be05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:48.590124 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.589988    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ebb4c011-6706-4815-83a0-8bccd981be05" (UID: "ebb4c011-6706-4815-83a0-8bccd981be05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:48.591798 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.591770    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb4c011-6706-4815-83a0-8bccd981be05-istio-token" (OuterVolumeSpecName: "istio-token") pod "ebb4c011-6706-4815-83a0-8bccd981be05" (UID: "ebb4c011-6706-4815-83a0-8bccd981be05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:48.592055 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.592019    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ebb4c011-6706-4815-83a0-8bccd981be05" (UID: "ebb4c011-6706-4815-83a0-8bccd981be05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:48.592142 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.592089    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb4c011-6706-4815-83a0-8bccd981be05-kube-api-access-76jfs" (OuterVolumeSpecName: "kube-api-access-76jfs") pod "ebb4c011-6706-4815-83a0-8bccd981be05" (UID: "ebb4c011-6706-4815-83a0-8bccd981be05"). InnerVolumeSpecName "kube-api-access-76jfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:48.592185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.592143    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebb4c011-6706-4815-83a0-8bccd981be05-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ebb4c011-6706-4815-83a0-8bccd981be05" (UID: "ebb4c011-6706-4815-83a0-8bccd981be05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:36:48.689261 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.689221    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:48.689261 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.689260    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:48.689533 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.689275    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:48.689533 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.689289    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76jfs\" (UniqueName: \"kubernetes.io/projected/ebb4c011-6706-4815-83a0-8bccd981be05-kube-api-access-76jfs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:48.689533 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.689304    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebb4c011-6706-4815-83a0-8bccd981be05-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:48.689533 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.689318    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebb4c011-6706-4815-83a0-8bccd981be05-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:48.689533 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.689330    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:48.689533 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.689344    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebb4c011-6706-4815-83a0-8bccd981be05-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:48.689533 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.689356    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebb4c011-6706-4815-83a0-8bccd981be05-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:48.753421 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.753327    2563 generic.go:356] "Generic (PLEG): container finished" podID="ebb4c011-6706-4815-83a0-8bccd981be05" containerID="cc11bddc036aa43df220623d56ad5744bbdc77eb65118ff77224df5dfe86a9d5" exitCode=0
Sep 16 21:36:48.753421 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.753376    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"
Sep 16 21:36:48.753875 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.753425    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x" event={"ID":"ebb4c011-6706-4815-83a0-8bccd981be05","Type":"ContainerDied","Data":"cc11bddc036aa43df220623d56ad5744bbdc77eb65118ff77224df5dfe86a9d5"}
Sep 16 21:36:48.753875 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.753471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x" event={"ID":"ebb4c011-6706-4815-83a0-8bccd981be05","Type":"ContainerDied","Data":"362b9824665c1c1579fd2be1dad7c544fc176c5f33f6ebae976d31594c22ba05"}
Sep 16 21:36:48.753875 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.753505    2563 scope.go:117] "RemoveContainer" containerID="cc11bddc036aa43df220623d56ad5744bbdc77eb65118ff77224df5dfe86a9d5"
Sep 16 21:36:48.762639 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.762609    2563 scope.go:117] "RemoveContainer" containerID="37f265530fc93bb27736d93122b9aec35b0dffa9c2af91d1e8d23e1d0db75ae6"
Sep 16 21:36:48.780863 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.780841    2563 scope.go:117] "RemoveContainer" containerID="cc11bddc036aa43df220623d56ad5744bbdc77eb65118ff77224df5dfe86a9d5"
Sep 16 21:36:48.781279 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:36:48.781208    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc11bddc036aa43df220623d56ad5744bbdc77eb65118ff77224df5dfe86a9d5\": container with ID starting with cc11bddc036aa43df220623d56ad5744bbdc77eb65118ff77224df5dfe86a9d5 not found: ID does not exist" containerID="cc11bddc036aa43df220623d56ad5744bbdc77eb65118ff77224df5dfe86a9d5"
Sep 16 21:36:48.781279 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.781254    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc11bddc036aa43df220623d56ad5744bbdc77eb65118ff77224df5dfe86a9d5"} err="failed to get container status \"cc11bddc036aa43df220623d56ad5744bbdc77eb65118ff77224df5dfe86a9d5\": rpc error: code = NotFound desc = could not find container \"cc11bddc036aa43df220623d56ad5744bbdc77eb65118ff77224df5dfe86a9d5\": container with ID starting with cc11bddc036aa43df220623d56ad5744bbdc77eb65118ff77224df5dfe86a9d5 not found: ID does not exist"
Sep 16 21:36:48.781279 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.781279    2563 scope.go:117] "RemoveContainer" containerID="37f265530fc93bb27736d93122b9aec35b0dffa9c2af91d1e8d23e1d0db75ae6"
Sep 16 21:36:48.781611 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:36:48.781583    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f265530fc93bb27736d93122b9aec35b0dffa9c2af91d1e8d23e1d0db75ae6\": container with ID starting with 37f265530fc93bb27736d93122b9aec35b0dffa9c2af91d1e8d23e1d0db75ae6 not found: ID does not exist" containerID="37f265530fc93bb27736d93122b9aec35b0dffa9c2af91d1e8d23e1d0db75ae6"
Sep 16 21:36:48.781705 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.781620    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f265530fc93bb27736d93122b9aec35b0dffa9c2af91d1e8d23e1d0db75ae6"} err="failed to get container status \"37f265530fc93bb27736d93122b9aec35b0dffa9c2af91d1e8d23e1d0db75ae6\": rpc error: code = NotFound desc = could not find container \"37f265530fc93bb27736d93122b9aec35b0dffa9c2af91d1e8d23e1d0db75ae6\": container with ID starting with 37f265530fc93bb27736d93122b9aec35b0dffa9c2af91d1e8d23e1d0db75ae6 not found: ID does not exist"
Sep 16 21:36:48.787414 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.787383    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"]
Sep 16 21:36:48.791737 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:48.790986    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-ddbefa-openshift-default-5cd48889cd-9hv6x"]
Sep 16 21:36:49.460748 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:49.460718    2563 patch_prober.go:28] interesting pod/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 16 21:36:49.460912 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:49.460771    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78" podUID="60c75e61-24ae-4d13-8a3c-ee68158b1f2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 16 21:36:50.460549 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:50.460516    2563 patch_prober.go:28] interesting pod/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 16 21:36:50.460957 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:50.460569    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78" podUID="60c75e61-24ae-4d13-8a3c-ee68158b1f2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 16 21:36:50.608813 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:50.608782    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebb4c011-6706-4815-83a0-8bccd981be05" path="/var/lib/kubelet/pods/ebb4c011-6706-4815-83a0-8bccd981be05/volumes"
Sep 16 21:36:51.460676 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:51.460509    2563 patch_prober.go:28] interesting pod/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 16 21:36:51.460676 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:51.460601    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78" podUID="60c75e61-24ae-4d13-8a3c-ee68158b1f2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 16 21:36:51.900165 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:51.900069    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"]
Sep 16 21:36:51.900331 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:51.900309    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78" podUID="60c75e61-24ae-4d13-8a3c-ee68158b1f2d" containerName="istio-proxy" containerID="cri-o://89829aef533fe975f94c958eed2ef6db884a6dea7b069c8cdcfd08ab074568a6" gracePeriod=30
Sep 16 21:36:58.154791 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.154767    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:58.192647 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.192622    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-data\") pod \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") "
Sep 16 21:36:58.192794 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.192677    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-podinfo\") pod \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") "
Sep 16 21:36:58.192794 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.192704    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rg5jd\" (UniqueName: \"kubernetes.io/projected/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-kube-api-access-rg5jd\") pod \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") "
Sep 16 21:36:58.192794 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.192729    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-envoy\") pod \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") "
Sep 16 21:36:58.192966 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.192794    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-token\") pod \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") "
Sep 16 21:36:58.192966 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.192830    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-workload-socket\") pod \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") "
Sep 16 21:36:58.192966 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.192855    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-workload-certs\") pod \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") "
Sep 16 21:36:58.192966 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.192884    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istiod-ca-cert\") pod \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") "
Sep 16 21:36:58.192966 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.192930    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-credential-socket\") pod \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\" (UID: \"60c75e61-24ae-4d13-8a3c-ee68158b1f2d\") "
Sep 16 21:36:58.193225 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.193099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "60c75e61-24ae-4d13-8a3c-ee68158b1f2d" (UID: "60c75e61-24ae-4d13-8a3c-ee68158b1f2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:58.193314 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.193290    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "60c75e61-24ae-4d13-8a3c-ee68158b1f2d" (UID: "60c75e61-24ae-4d13-8a3c-ee68158b1f2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:58.193377 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.193337    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:58.193449 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.193410    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-data" (OuterVolumeSpecName: "istio-data") pod "60c75e61-24ae-4d13-8a3c-ee68158b1f2d" (UID: "60c75e61-24ae-4d13-8a3c-ee68158b1f2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:58.193505 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.193435    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "60c75e61-24ae-4d13-8a3c-ee68158b1f2d" (UID: "60c75e61-24ae-4d13-8a3c-ee68158b1f2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:58.193619 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.193594    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "60c75e61-24ae-4d13-8a3c-ee68158b1f2d" (UID: "60c75e61-24ae-4d13-8a3c-ee68158b1f2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:36:58.195020 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.194993    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-kube-api-access-rg5jd" (OuterVolumeSpecName: "kube-api-access-rg5jd") pod "60c75e61-24ae-4d13-8a3c-ee68158b1f2d" (UID: "60c75e61-24ae-4d13-8a3c-ee68158b1f2d"). InnerVolumeSpecName "kube-api-access-rg5jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:58.195515 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.195493    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "60c75e61-24ae-4d13-8a3c-ee68158b1f2d" (UID: "60c75e61-24ae-4d13-8a3c-ee68158b1f2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:36:58.195574 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.195537    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-token" (OuterVolumeSpecName: "istio-token") pod "60c75e61-24ae-4d13-8a3c-ee68158b1f2d" (UID: "60c75e61-24ae-4d13-8a3c-ee68158b1f2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:58.195574 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.195544    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "60c75e61-24ae-4d13-8a3c-ee68158b1f2d" (UID: "60c75e61-24ae-4d13-8a3c-ee68158b1f2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:36:58.293882 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.293818    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:58.293882 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.293843    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:58.293882 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.293853    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:58.293882 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.293861    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:58.293882 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.293869    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg5jd\" (UniqueName: \"kubernetes.io/projected/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-kube-api-access-rg5jd\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:58.293882 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.293877    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:58.293882 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.293885    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:58.294164 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.293892    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c75e61-24ae-4d13-8a3c-ee68158b1f2d-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:36:58.813911 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.813877    2563 generic.go:356] "Generic (PLEG): container finished" podID="60c75e61-24ae-4d13-8a3c-ee68158b1f2d" containerID="89829aef533fe975f94c958eed2ef6db884a6dea7b069c8cdcfd08ab074568a6" exitCode=0
Sep 16 21:36:58.814048 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.813923    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"
Sep 16 21:36:58.814048 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.813940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78" event={"ID":"60c75e61-24ae-4d13-8a3c-ee68158b1f2d","Type":"ContainerDied","Data":"89829aef533fe975f94c958eed2ef6db884a6dea7b069c8cdcfd08ab074568a6"}
Sep 16 21:36:58.814048 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.813982    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78" event={"ID":"60c75e61-24ae-4d13-8a3c-ee68158b1f2d","Type":"ContainerDied","Data":"213cd136230db4989944a4a37483f0cc413952c5c3aacce8a469b4fbc4ae0a88"}
Sep 16 21:36:58.814048 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.814004    2563 scope.go:117] "RemoveContainer" containerID="89829aef533fe975f94c958eed2ef6db884a6dea7b069c8cdcfd08ab074568a6"
Sep 16 21:36:58.823013 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.822994    2563 scope.go:117] "RemoveContainer" containerID="a0bf9670658bf5115e5e6b4c309c3f034cdda5f00c1a51312d5394fed095d289"
Sep 16 21:36:58.831015 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.830996    2563 scope.go:117] "RemoveContainer" containerID="89829aef533fe975f94c958eed2ef6db884a6dea7b069c8cdcfd08ab074568a6"
Sep 16 21:36:58.831265 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:36:58.831247    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89829aef533fe975f94c958eed2ef6db884a6dea7b069c8cdcfd08ab074568a6\": container with ID starting with 89829aef533fe975f94c958eed2ef6db884a6dea7b069c8cdcfd08ab074568a6 not found: ID does not exist" containerID="89829aef533fe975f94c958eed2ef6db884a6dea7b069c8cdcfd08ab074568a6"
Sep 16 21:36:58.831333 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.831275    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89829aef533fe975f94c958eed2ef6db884a6dea7b069c8cdcfd08ab074568a6"} err="failed to get container status \"89829aef533fe975f94c958eed2ef6db884a6dea7b069c8cdcfd08ab074568a6\": rpc error: code = NotFound desc = could not find container \"89829aef533fe975f94c958eed2ef6db884a6dea7b069c8cdcfd08ab074568a6\": container with ID starting with 89829aef533fe975f94c958eed2ef6db884a6dea7b069c8cdcfd08ab074568a6 not found: ID does not exist"
Sep 16 21:36:58.831333 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.831293    2563 scope.go:117] "RemoveContainer" containerID="a0bf9670658bf5115e5e6b4c309c3f034cdda5f00c1a51312d5394fed095d289"
Sep 16 21:36:58.831531 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:36:58.831513    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0bf9670658bf5115e5e6b4c309c3f034cdda5f00c1a51312d5394fed095d289\": container with ID starting with a0bf9670658bf5115e5e6b4c309c3f034cdda5f00c1a51312d5394fed095d289 not found: ID does not exist" containerID="a0bf9670658bf5115e5e6b4c309c3f034cdda5f00c1a51312d5394fed095d289"
Sep 16 21:36:58.831593 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.831537    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bf9670658bf5115e5e6b4c309c3f034cdda5f00c1a51312d5394fed095d289"} err="failed to get container status \"a0bf9670658bf5115e5e6b4c309c3f034cdda5f00c1a51312d5394fed095d289\": rpc error: code = NotFound desc = could not find container \"a0bf9670658bf5115e5e6b4c309c3f034cdda5f00c1a51312d5394fed095d289\": container with ID starting with a0bf9670658bf5115e5e6b4c309c3f034cdda5f00c1a51312d5394fed095d289 not found: ID does not exist"
Sep 16 21:36:58.831593 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.831573    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"]
Sep 16 21:36:58.835265 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:36:58.835245    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a1fb6d89-openshift-default-7d797dd4b4-hft78"]
Sep 16 21:37:00.608963 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:00.608929    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60c75e61-24ae-4d13-8a3c-ee68158b1f2d" path="/var/lib/kubelet/pods/60c75e61-24ae-4d13-8a3c-ee68158b1f2d/volumes"
Sep 16 21:37:11.886700 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:11.886666    2563 generic.go:356] "Generic (PLEG): container finished" podID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" containerID="34110ba7cc805bcd8f5429e8cc6232bf95c4bc2997bffda85ccf24f54b311950" exitCode=0
Sep 16 21:37:11.886700 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:11.886684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82" event={"ID":"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a","Type":"ContainerDied","Data":"34110ba7cc805bcd8f5429e8cc6232bf95c4bc2997bffda85ccf24f54b311950"}
Sep 16 21:37:11.887133 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:11.887103    2563 scope.go:117] "RemoveContainer" containerID="34110ba7cc805bcd8f5429e8cc6232bf95c4bc2997bffda85ccf24f54b311950"
Sep 16 21:37:11.888276 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:11.888244    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:37:11.888384 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:11.888331    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:37:11.888456 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:11.888385    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:37:12.895635 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:12.895603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82" event={"ID":"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a","Type":"ContainerStarted","Data":"9daccf14e57e354e5cc52ab7dfead179a4bf79ab10e5cb8465222541599f088e"}
Sep 16 21:37:13.249581 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:13.249537    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:37:13.250874 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:13.250848    2563 patch_prober.go:28] interesting pod/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 16 21:37:13.250970 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:13.250901    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82" podUID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 16 21:37:14.249107 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:14.249070    2563 patch_prober.go:28] interesting pod/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 16 21:37:14.249513 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:14.249141    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82" podUID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 16 21:37:15.249674 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:15.249615    2563 patch_prober.go:28] interesting pod/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 16 21:37:15.250055 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:15.249711    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82" podUID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 16 21:37:15.733010 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:15.732971    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:37:16.249776 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:16.249741    2563 patch_prober.go:28] interesting pod/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 16 21:37:16.250173 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:16.249807    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82" podUID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 16 21:37:17.248856 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:17.248817    2563 patch_prober.go:28] interesting pod/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 16 21:37:17.249044 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:17.248878    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82" podUID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 16 21:37:18.254081 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:18.254050    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:37:18.255204 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:18.255185    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:37:26.980018 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.979983    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"]
Sep 16 21:37:26.980512 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.980497    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb4c011-6706-4815-83a0-8bccd981be05" containerName="istio-proxy"
Sep 16 21:37:26.980563 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.980515    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb4c011-6706-4815-83a0-8bccd981be05" containerName="istio-proxy"
Sep 16 21:37:26.980563 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.980528    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c75e61-24ae-4d13-8a3c-ee68158b1f2d" containerName="istio-proxy"
Sep 16 21:37:26.980563 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.980537    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c75e61-24ae-4d13-8a3c-ee68158b1f2d" containerName="istio-proxy"
Sep 16 21:37:26.980563 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.980552    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c75e61-24ae-4d13-8a3c-ee68158b1f2d" containerName="istio-proxy"
Sep 16 21:37:26.980563 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.980562    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c75e61-24ae-4d13-8a3c-ee68158b1f2d" containerName="istio-proxy"
Sep 16 21:37:26.980746 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.980666    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c75e61-24ae-4d13-8a3c-ee68158b1f2d" containerName="istio-proxy"
Sep 16 21:37:26.980746 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.980677    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb4c011-6706-4815-83a0-8bccd981be05" containerName="istio-proxy"
Sep 16 21:37:26.980746 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.980691    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c75e61-24ae-4d13-8a3c-ee68158b1f2d" containerName="istio-proxy"
Sep 16 21:37:26.985587 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.985566    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:26.987846 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.987814    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-44e517\"/\"openshift-service-ca.crt\""
Sep 16 21:37:26.988913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.988526    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-44e517\"/\"test-model-44e517-kserve-self-signed-certs\""
Sep 16 21:37:26.988913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.988630    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-44e517\"/\"kube-root-ca.crt\""
Sep 16 21:37:26.988913 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.988871    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-44e517\"/\"default-dockercfg-95bjl\""
Sep 16 21:37:26.991774 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:26.991751    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"]
Sep 16 21:37:27.006360 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.006337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-tmp-dir\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.006491 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.006370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-model-cache\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.006491 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.006462    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-dshm\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.006601 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.006497    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-home\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.006715 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.006604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr5gm\" (UniqueName: \"kubernetes.io/projected/68bb534f-c799-4262-85fc-4b3ad900c1ca-kube-api-access-fr5gm\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.006715 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.006694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bb534f-c799-4262-85fc-4b3ad900c1ca-tls-certs\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.107895 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.107862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr5gm\" (UniqueName: \"kubernetes.io/projected/68bb534f-c799-4262-85fc-4b3ad900c1ca-kube-api-access-fr5gm\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.108089 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.107918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bb534f-c799-4262-85fc-4b3ad900c1ca-tls-certs\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.108089 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.107989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-tmp-dir\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.108089 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.108023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-model-cache\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.108252 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.108090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-dshm\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.108252 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.108116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-home\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.108458 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.108435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-tmp-dir\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.108519 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.108487    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-model-cache\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.108576 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.108536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-home\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.110273 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.110251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-dshm\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.110806 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.110790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bb534f-c799-4262-85fc-4b3ad900c1ca-tls-certs\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.115202 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.115178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr5gm\" (UniqueName: \"kubernetes.io/projected/68bb534f-c799-4262-85fc-4b3ad900c1ca-kube-api-access-fr5gm\") pod \"test-model-44e517-kserve-c5db6b68f-hq7g5\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") " pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.297916 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.297832    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:27.574139 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.574108    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"]
Sep 16 21:37:27.575511 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:37:27.575480    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68bb534f_c799_4262_85fc_4b3ad900c1ca.slice/crio-41c2b16768374c238b95cade3e665cb6f09442bce8fc2fcd7fed820ba22725cf WatchSource:0}: Error finding container 41c2b16768374c238b95cade3e665cb6f09442bce8fc2fcd7fed820ba22725cf: Status 404 returned error can't find the container with id 41c2b16768374c238b95cade3e665cb6f09442bce8fc2fcd7fed820ba22725cf
Sep 16 21:37:27.977727 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.977691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5" event={"ID":"68bb534f-c799-4262-85fc-4b3ad900c1ca","Type":"ContainerStarted","Data":"e428561b5c69b166bb292d32305e331abaabeb7d5174397b6438911184bfa892"}
Sep 16 21:37:27.977727 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.977739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5" event={"ID":"68bb534f-c799-4262-85fc-4b3ad900c1ca","Type":"ContainerStarted","Data":"41c2b16768374c238b95cade3e665cb6f09442bce8fc2fcd7fed820ba22725cf"}
Sep 16 21:37:27.996552 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:27.996504    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5" podStartSLOduration=1.996488549 podStartE2EDuration="1.996488549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:37:27.995003701 +0000 UTC m=+907.895213706" watchObservedRunningTime="2026-09-16 21:37:27.996488549 +0000 UTC m=+907.896698552"
Sep 16 21:37:37.299043 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:37.298994    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:37.299544 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:37.299065    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:37.311614 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:37.311588    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:37:38.041042 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:37:38.041008    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:38:14.469737 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:14.469702    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"]
Sep 16 21:38:14.470218 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:14.469977    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82" podUID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" containerName="istio-proxy" containerID="cri-o://9daccf14e57e354e5cc52ab7dfead179a4bf79ab10e5cb8465222541599f088e" gracePeriod=30
Sep 16 21:38:15.232694 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.232646    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"]
Sep 16 21:38:15.233134 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.233114    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb4c011-6706-4815-83a0-8bccd981be05" containerName="istio-proxy"
Sep 16 21:38:15.233216 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.233139    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb4c011-6706-4815-83a0-8bccd981be05" containerName="istio-proxy"
Sep 16 21:38:15.233280 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.233269    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb4c011-6706-4815-83a0-8bccd981be05" containerName="istio-proxy"
Sep 16 21:38:15.236645 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.236627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.238977 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.238954    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-0a22a7-openshift-default-dockercfg-8b7jt\""
Sep 16 21:38:15.248831 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.248809    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"]
Sep 16 21:38:15.287280 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.287251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-envoy\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.287412 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.287290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-data\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.287412 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.287310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-workload-certs\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.287412 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.287357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-token\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.287580 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.287431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-credential-socket\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.287580 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.287499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-podinfo\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.287580 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.287545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gbw5\" (UniqueName: \"kubernetes.io/projected/81f62c3e-ce8a-4860-acfb-d50aff0821cb-kube-api-access-7gbw5\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.287735 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.287586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-workload-socket\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.287735 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.287604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istiod-ca-cert\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.388622 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.388589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-envoy\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.388814 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.388634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-data\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.388814 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.388686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-workload-certs\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.388814 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.388714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-token\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.388980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.388863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-credential-socket\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.388980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.388958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-podinfo\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.389101 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.389014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbw5\" (UniqueName: \"kubernetes.io/projected/81f62c3e-ce8a-4860-acfb-d50aff0821cb-kube-api-access-7gbw5\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.389101 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.389068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-workload-socket\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.389101 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.389076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-data\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.389258 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.389159    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-workload-certs\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.389258 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.389211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istiod-ca-cert\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.389258 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.389223    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-credential-socket\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.389413 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.389377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-workload-socket\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.389914 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.389882    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istiod-ca-cert\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.391028 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.391002    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-envoy\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.391390 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.391374    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-podinfo\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.402254 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.402228    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-token\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.402545 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.402525    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gbw5\" (UniqueName: \"kubernetes.io/projected/81f62c3e-ce8a-4860-acfb-d50aff0821cb-kube-api-access-7gbw5\") pod \"e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") " pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.548221 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.548145    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:15.822302 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.822219    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"]
Sep 16 21:38:15.825569 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:38:15.825543    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f62c3e_ce8a_4860_acfb_d50aff0821cb.slice/crio-5fb29c1f7e2f1bdc074f74dd9fa55dbd3c5b0b0d35c456e2747f8e76f487add7 WatchSource:0}: Error finding container 5fb29c1f7e2f1bdc074f74dd9fa55dbd3c5b0b0d35c456e2747f8e76f487add7: Status 404 returned error can't find the container with id 5fb29c1f7e2f1bdc074f74dd9fa55dbd3c5b0b0d35c456e2747f8e76f487add7
Sep 16 21:38:15.827803 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.827769    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:38:15.827906 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.827832    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:38:15.827906 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:15.827877    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:38:16.222999 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:16.222963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm" event={"ID":"81f62c3e-ce8a-4860-acfb-d50aff0821cb","Type":"ContainerStarted","Data":"75ffef74499ddc1d0c7f3c8fc2967da552ae9cc1dbb3eb2ceebc8f758ecae6a7"}
Sep 16 21:38:16.223154 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:16.223013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm" event={"ID":"81f62c3e-ce8a-4860-acfb-d50aff0821cb","Type":"ContainerStarted","Data":"5fb29c1f7e2f1bdc074f74dd9fa55dbd3c5b0b0d35c456e2747f8e76f487add7"}
Sep 16 21:38:16.245203 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:16.245156    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm" podStartSLOduration=1.245139719 podStartE2EDuration="1.245139719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:38:16.243442908 +0000 UTC m=+956.143652913" watchObservedRunningTime="2026-09-16 21:38:16.245139719 +0000 UTC m=+956.145349808"
Sep 16 21:38:16.548768 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:16.548694    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:16.553897 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:16.553876    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:17.228365 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:17.228332    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:17.229364 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:17.229345    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:38:18.256376 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:18.256345    2563 patch_prober.go:28] interesting pod/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 21:38:18.256761 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:18.256398    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82" podUID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 21:38:19.725245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.725216    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:38:19.836451 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.836361    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-workload-socket\") pod \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") "
Sep 16 21:38:19.836451 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.836402    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-workload-certs\") pod \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") "
Sep 16 21:38:19.836451 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.836442    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-envoy\") pod \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") "
Sep 16 21:38:19.836706 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.836526    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-credential-socket\") pod \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") "
Sep 16 21:38:19.836706 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.836584    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gzml\" (UniqueName: \"kubernetes.io/projected/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-kube-api-access-8gzml\") pod \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") "
Sep 16 21:38:19.836706 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.836673    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" (UID: "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:38:19.836841 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.836710    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-data\") pod \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") "
Sep 16 21:38:19.836841 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.836753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istiod-ca-cert\") pod \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") "
Sep 16 21:38:19.836841 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.836792    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-podinfo\") pod \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") "
Sep 16 21:38:19.836841 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.836815    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-token\") pod \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\" (UID: \"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a\") "
Sep 16 21:38:19.836841 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.836827    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" (UID: "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:38:19.837104 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.836810    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" (UID: "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:38:19.837104 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.836997    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-data" (OuterVolumeSpecName: "istio-data") pod "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" (UID: "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:38:19.837206 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.837135    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" (UID: "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:38:19.837250 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.837238    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:38:19.837305 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.837257    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:38:19.837305 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.837270    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:38:19.837305 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.837283    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:38:19.837305 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.837296    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:38:19.838845 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.838815    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" (UID: "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:38:19.838965 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.838857    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" (UID: "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:38:19.838965 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.838898    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-token" (OuterVolumeSpecName: "istio-token") pod "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" (UID: "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:38:19.838965 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.838925    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-kube-api-access-8gzml" (OuterVolumeSpecName: "kube-api-access-8gzml") pod "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" (UID: "8576f11f-bd1e-4c61-8a0f-fdac1fe4467a"). InnerVolumeSpecName "kube-api-access-8gzml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:38:19.938812 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.938775    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:38:19.938812 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.938813    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:38:19.939068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.938828    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:38:19.939068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:19.938843    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gzml\" (UniqueName: \"kubernetes.io/projected/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a-kube-api-access-8gzml\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:38:20.247427 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:20.247395    2563 generic.go:356] "Generic (PLEG): container finished" podID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" containerID="9daccf14e57e354e5cc52ab7dfead179a4bf79ab10e5cb8465222541599f088e" exitCode=0
Sep 16 21:38:20.247591 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:20.247441    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"
Sep 16 21:38:20.247591 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:20.247459    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82" event={"ID":"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a","Type":"ContainerDied","Data":"9daccf14e57e354e5cc52ab7dfead179a4bf79ab10e5cb8465222541599f088e"}
Sep 16 21:38:20.247591 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:20.247503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82" event={"ID":"8576f11f-bd1e-4c61-8a0f-fdac1fe4467a","Type":"ContainerDied","Data":"4704ba08e1d81862682453bc1989e472948afd46e87206d6d30ae7fc5bb4d62e"}
Sep 16 21:38:20.247591 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:20.247524    2563 scope.go:117] "RemoveContainer" containerID="9daccf14e57e354e5cc52ab7dfead179a4bf79ab10e5cb8465222541599f088e"
Sep 16 21:38:20.257498 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:20.257478    2563 scope.go:117] "RemoveContainer" containerID="34110ba7cc805bcd8f5429e8cc6232bf95c4bc2997bffda85ccf24f54b311950"
Sep 16 21:38:20.265207 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:20.265191    2563 scope.go:117] "RemoveContainer" containerID="9daccf14e57e354e5cc52ab7dfead179a4bf79ab10e5cb8465222541599f088e"
Sep 16 21:38:20.265460 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:38:20.265440    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9daccf14e57e354e5cc52ab7dfead179a4bf79ab10e5cb8465222541599f088e\": container with ID starting with 9daccf14e57e354e5cc52ab7dfead179a4bf79ab10e5cb8465222541599f088e not found: ID does not exist" containerID="9daccf14e57e354e5cc52ab7dfead179a4bf79ab10e5cb8465222541599f088e"
Sep 16 21:38:20.265539 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:20.265473    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9daccf14e57e354e5cc52ab7dfead179a4bf79ab10e5cb8465222541599f088e"} err="failed to get container status \"9daccf14e57e354e5cc52ab7dfead179a4bf79ab10e5cb8465222541599f088e\": rpc error: code = NotFound desc = could not find container \"9daccf14e57e354e5cc52ab7dfead179a4bf79ab10e5cb8465222541599f088e\": container with ID starting with 9daccf14e57e354e5cc52ab7dfead179a4bf79ab10e5cb8465222541599f088e not found: ID does not exist"
Sep 16 21:38:20.265539 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:20.265496    2563 scope.go:117] "RemoveContainer" containerID="34110ba7cc805bcd8f5429e8cc6232bf95c4bc2997bffda85ccf24f54b311950"
Sep 16 21:38:20.265824 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:38:20.265805    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34110ba7cc805bcd8f5429e8cc6232bf95c4bc2997bffda85ccf24f54b311950\": container with ID starting with 34110ba7cc805bcd8f5429e8cc6232bf95c4bc2997bffda85ccf24f54b311950 not found: ID does not exist" containerID="34110ba7cc805bcd8f5429e8cc6232bf95c4bc2997bffda85ccf24f54b311950"
Sep 16 21:38:20.265891 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:20.265832    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34110ba7cc805bcd8f5429e8cc6232bf95c4bc2997bffda85ccf24f54b311950"} err="failed to get container status \"34110ba7cc805bcd8f5429e8cc6232bf95c4bc2997bffda85ccf24f54b311950\": rpc error: code = NotFound desc = could not find container \"34110ba7cc805bcd8f5429e8cc6232bf95c4bc2997bffda85ccf24f54b311950\": container with ID starting with 34110ba7cc805bcd8f5429e8cc6232bf95c4bc2997bffda85ccf24f54b311950 not found: ID does not exist"
Sep 16 21:38:20.269053 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:20.269031    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"]
Sep 16 21:38:20.272550 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:20.272531    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-b91829-openshift-default-65d8cc6bcb-dch82"]
Sep 16 21:38:20.614402 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:20.614324    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" path="/var/lib/kubelet/pods/8576f11f-bd1e-4c61-8a0f-fdac1fe4467a/volumes"
Sep 16 21:38:27.645838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.645797    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"]
Sep 16 21:38:27.646353 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.646332    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" containerName="istio-proxy"
Sep 16 21:38:27.646393 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.646357    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" containerName="istio-proxy"
Sep 16 21:38:27.646444 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.646401    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" containerName="istio-proxy"
Sep 16 21:38:27.646444 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.646408    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" containerName="istio-proxy"
Sep 16 21:38:27.646527 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.646513    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" containerName="istio-proxy"
Sep 16 21:38:27.646573 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.646536    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8576f11f-bd1e-4c61-8a0f-fdac1fe4467a" containerName="istio-proxy"
Sep 16 21:38:27.651035 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.651012    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.653112 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.653092    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-06ac85db-openshift-default-dockercfg-mq85m\""
Sep 16 21:38:27.660833 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.660811    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"]
Sep 16 21:38:27.821273 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.821232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-workload-socket\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.821467 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.821285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-credential-socket\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.821467 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.821324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2984g\" (UniqueName: \"kubernetes.io/projected/1b335023-b1b8-4e94-888d-f8477c225d58-kube-api-access-2984g\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.821467 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.821385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b335023-b1b8-4e94-888d-f8477c225d58-istio-token\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.821467 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.821413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b335023-b1b8-4e94-888d-f8477c225d58-istiod-ca-cert\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.821467 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.821446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-istio-envoy\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.821825 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.821490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-istio-data\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.821825 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.821517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-workload-certs\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.821825 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.821572    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b335023-b1b8-4e94-888d-f8477c225d58-istio-podinfo\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.923031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.923003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-workload-certs\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.923218 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.923117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b335023-b1b8-4e94-888d-f8477c225d58-istio-podinfo\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.923270 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.923230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-workload-socket\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.923320 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.923268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-credential-socket\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.923320 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.923310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2984g\" (UniqueName: \"kubernetes.io/projected/1b335023-b1b8-4e94-888d-f8477c225d58-kube-api-access-2984g\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.923431 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.923361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-workload-certs\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.923431 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.923381    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b335023-b1b8-4e94-888d-f8477c225d58-istio-token\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.923431 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.923419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b335023-b1b8-4e94-888d-f8477c225d58-istiod-ca-cert\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.923592 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.923449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-istio-envoy\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.923592 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.923494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-istio-data\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.923726 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.923606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-workload-socket\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.923726 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.923629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-credential-socket\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.924087 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.924062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-istio-data\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.924254 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.924235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b335023-b1b8-4e94-888d-f8477c225d58-istiod-ca-cert\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.925755 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.925731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-istio-envoy\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.925858 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.925843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b335023-b1b8-4e94-888d-f8477c225d58-istio-podinfo\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.931141 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.931121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b335023-b1b8-4e94-888d-f8477c225d58-istio-token\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.931540 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.931518    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2984g\" (UniqueName: \"kubernetes.io/projected/1b335023-b1b8-4e94-888d-f8477c225d58-kube-api-access-2984g\") pod \"e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") " pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:27.962634 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:27.962604    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:28.290766 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:28.290735    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"]
Sep 16 21:38:28.292694 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:38:28.292647    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b335023_b1b8_4e94_888d_f8477c225d58.slice/crio-079b4cd4be43fb300fe58cef817c0f4de3a4d503ae99d17f806e3d30453bf430 WatchSource:0}: Error finding container 079b4cd4be43fb300fe58cef817c0f4de3a4d503ae99d17f806e3d30453bf430: Status 404 returned error can't find the container with id 079b4cd4be43fb300fe58cef817c0f4de3a4d503ae99d17f806e3d30453bf430
Sep 16 21:38:28.295053 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:28.295018    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:38:28.295157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:28.295096    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:38:28.295157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:28.295127    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:38:29.294884 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:29.294839    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl" event={"ID":"1b335023-b1b8-4e94-888d-f8477c225d58","Type":"ContainerStarted","Data":"b19f9f71d3807d7fe51d40974235260e969e1b4540ad6dd843af18defbdcf6d7"}
Sep 16 21:38:29.295300 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:29.294905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl" event={"ID":"1b335023-b1b8-4e94-888d-f8477c225d58","Type":"ContainerStarted","Data":"079b4cd4be43fb300fe58cef817c0f4de3a4d503ae99d17f806e3d30453bf430"}
Sep 16 21:38:29.345686 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:29.345617    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl" podStartSLOduration=2.3456016379999998 podStartE2EDuration="2.345601638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:38:29.344463682 +0000 UTC m=+969.244673687" watchObservedRunningTime="2026-09-16 21:38:29.345601638 +0000 UTC m=+969.245811642"
Sep 16 21:38:29.963537 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:29.963498    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:29.968383 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:29.968357    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:30.300396 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:30.300309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:38:30.301161 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:38:30.301142    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:39:04.482089 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:04.481999    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b335023-b1b8-4e94-888d-f8477c225d58" containerID="b19f9f71d3807d7fe51d40974235260e969e1b4540ad6dd843af18defbdcf6d7" exitCode=0
Sep 16 21:39:04.482089 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:04.482047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl" event={"ID":"1b335023-b1b8-4e94-888d-f8477c225d58","Type":"ContainerDied","Data":"b19f9f71d3807d7fe51d40974235260e969e1b4540ad6dd843af18defbdcf6d7"}
Sep 16 21:39:04.482514 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:04.482477    2563 scope.go:117] "RemoveContainer" containerID="b19f9f71d3807d7fe51d40974235260e969e1b4540ad6dd843af18defbdcf6d7"
Sep 16 21:39:04.483704 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:04.483648    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:39:04.483796 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:04.483736    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:39:04.483796 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:04.483769    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:39:05.495844 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:05.495807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl" event={"ID":"1b335023-b1b8-4e94-888d-f8477c225d58","Type":"ContainerStarted","Data":"275d6c32183476b309edae7bb06d326a0841594700e8dadc0de78bd8dbd3d57c"}
Sep 16 21:39:05.963412 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:05.963372    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:39:05.963624 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:05.963599    2563 patch_prober.go:28] interesting pod/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 16 21:39:05.963729 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:05.963650    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl" podUID="1b335023-b1b8-4e94-888d-f8477c225d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 16 21:39:06.963790 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:06.963754    2563 patch_prober.go:28] interesting pod/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 16 21:39:06.964206 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:06.963816    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl" podUID="1b335023-b1b8-4e94-888d-f8477c225d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 16 21:39:07.962999 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:07.962961    2563 patch_prober.go:28] interesting pod/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 16 21:39:07.963211 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:07.963025    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl" podUID="1b335023-b1b8-4e94-888d-f8477c225d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 16 21:39:08.963289 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:08.963247    2563 patch_prober.go:28] interesting pod/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 16 21:39:08.963685 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:08.963320    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl" podUID="1b335023-b1b8-4e94-888d-f8477c225d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 16 21:39:09.963584 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:09.963548    2563 patch_prober.go:28] interesting pod/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 16 21:39:09.964033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:09.963625    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl" podUID="1b335023-b1b8-4e94-888d-f8477c225d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 16 21:39:10.967303 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:10.967272    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:39:10.967758 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:10.967592    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:39:10.968260 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:10.968240    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:39:11.366769 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:11.366676    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k"]
Sep 16 21:39:11.370319 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:11.370296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k"
Sep 16 21:39:11.372588 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:11.372565    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:39:11.372712 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:11.372568    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:39:11.378250 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:11.377899    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k"]
Sep 16 21:39:11.492518 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:11.492479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/595f9075-eaa3-4bb6-9374-8c1ac173ed8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k\" (UID: \"595f9075-eaa3-4bb6-9374-8c1ac173ed8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k"
Sep 16 21:39:11.593684 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:11.593610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/595f9075-eaa3-4bb6-9374-8c1ac173ed8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k\" (UID: \"595f9075-eaa3-4bb6-9374-8c1ac173ed8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k"
Sep 16 21:39:11.594271 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:11.594247    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/595f9075-eaa3-4bb6-9374-8c1ac173ed8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k\" (UID: \"595f9075-eaa3-4bb6-9374-8c1ac173ed8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k"
Sep 16 21:39:11.682708 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:11.682680    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k"
Sep 16 21:39:11.952579 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:11.952555    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k"]
Sep 16 21:39:11.954618 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:39:11.954594    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595f9075_eaa3_4bb6_9374_8c1ac173ed8f.slice/crio-3b05e2fba1db91662e1aa994022a0728f6ef28cb898f31ba0ee1b17715c21dd5 WatchSource:0}: Error finding container 3b05e2fba1db91662e1aa994022a0728f6ef28cb898f31ba0ee1b17715c21dd5: Status 404 returned error can't find the container with id 3b05e2fba1db91662e1aa994022a0728f6ef28cb898f31ba0ee1b17715c21dd5
Sep 16 21:39:12.542164 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:12.542071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k" event={"ID":"595f9075-eaa3-4bb6-9374-8c1ac173ed8f","Type":"ContainerStarted","Data":"ea98e5f1dfa6bda4a01f49cf896a7b7c0b7f6a79dfd084b7d227555c95d6f804"}
Sep 16 21:39:12.542164 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:12.542149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k" event={"ID":"595f9075-eaa3-4bb6-9374-8c1ac173ed8f","Type":"ContainerStarted","Data":"3b05e2fba1db91662e1aa994022a0728f6ef28cb898f31ba0ee1b17715c21dd5"}
Sep 16 21:39:12.556284 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:12.556217    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k" podStartSLOduration=1.556198136 podStartE2EDuration="1.556198136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:39:12.555843354 +0000 UTC m=+1012.456053363" watchObservedRunningTime="2026-09-16 21:39:12.556198136 +0000 UTC m=+1012.456408139"
Sep 16 21:39:13.549154 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:13.549116    2563 generic.go:356] "Generic (PLEG): container finished" podID="595f9075-eaa3-4bb6-9374-8c1ac173ed8f" containerID="ea98e5f1dfa6bda4a01f49cf896a7b7c0b7f6a79dfd084b7d227555c95d6f804" exitCode=0
Sep 16 21:39:13.549564 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:13.549199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k" event={"ID":"595f9075-eaa3-4bb6-9374-8c1ac173ed8f","Type":"ContainerDied","Data":"ea98e5f1dfa6bda4a01f49cf896a7b7c0b7f6a79dfd084b7d227555c95d6f804"}
Sep 16 21:39:14.693702 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:14.693678    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k"
Sep 16 21:39:14.827444 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:14.827326    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/595f9075-eaa3-4bb6-9374-8c1ac173ed8f-maas-api-service-ca\") pod \"595f9075-eaa3-4bb6-9374-8c1ac173ed8f\" (UID: \"595f9075-eaa3-4bb6-9374-8c1ac173ed8f\") "
Sep 16 21:39:14.827781 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:14.827751    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595f9075-eaa3-4bb6-9374-8c1ac173ed8f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "595f9075-eaa3-4bb6-9374-8c1ac173ed8f" (UID: "595f9075-eaa3-4bb6-9374-8c1ac173ed8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:39:14.827908 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:14.827893    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/595f9075-eaa3-4bb6-9374-8c1ac173ed8f-maas-api-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:15.562774 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:15.562690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k"
Sep 16 21:39:15.562774 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:15.562707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k" event={"ID":"595f9075-eaa3-4bb6-9374-8c1ac173ed8f","Type":"ContainerDied","Data":"3b05e2fba1db91662e1aa994022a0728f6ef28cb898f31ba0ee1b17715c21dd5"}
Sep 16 21:39:15.562774 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:15.562757    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b05e2fba1db91662e1aa994022a0728f6ef28cb898f31ba0ee1b17715c21dd5"
Sep 16 21:39:22.044844 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:22.044803    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k"]
Sep 16 21:39:22.046836 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:22.046823    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06ac85db-hhd2k"]
Sep 16 21:39:22.608258 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:22.608225    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="595f9075-eaa3-4bb6-9374-8c1ac173ed8f" path="/var/lib/kubelet/pods/595f9075-eaa3-4bb6-9374-8c1ac173ed8f/volumes"
Sep 16 21:39:28.488623 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:28.488587    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"]
Sep 16 21:39:28.489080 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:28.488855    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl" podUID="1b335023-b1b8-4e94-888d-f8477c225d58" containerName="istio-proxy" containerID="cri-o://275d6c32183476b309edae7bb06d326a0841594700e8dadc0de78bd8dbd3d57c" gracePeriod=30
Sep 16 21:39:33.659962 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.659925    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b335023-b1b8-4e94-888d-f8477c225d58" containerID="275d6c32183476b309edae7bb06d326a0841594700e8dadc0de78bd8dbd3d57c" exitCode=0
Sep 16 21:39:33.660735 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.659957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl" event={"ID":"1b335023-b1b8-4e94-888d-f8477c225d58","Type":"ContainerDied","Data":"275d6c32183476b309edae7bb06d326a0841594700e8dadc0de78bd8dbd3d57c"}
Sep 16 21:39:33.660735 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.660054    2563 scope.go:117] "RemoveContainer" containerID="b19f9f71d3807d7fe51d40974235260e969e1b4540ad6dd843af18defbdcf6d7"
Sep 16 21:39:33.744031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.744007    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:39:33.844477 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.844388    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-istio-data\") pod \"1b335023-b1b8-4e94-888d-f8477c225d58\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") "
Sep 16 21:39:33.844477 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.844435    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2984g\" (UniqueName: \"kubernetes.io/projected/1b335023-b1b8-4e94-888d-f8477c225d58-kube-api-access-2984g\") pod \"1b335023-b1b8-4e94-888d-f8477c225d58\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") "
Sep 16 21:39:33.844477 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.844464    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b335023-b1b8-4e94-888d-f8477c225d58-istio-token\") pod \"1b335023-b1b8-4e94-888d-f8477c225d58\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") "
Sep 16 21:39:33.844790 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.844486    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b335023-b1b8-4e94-888d-f8477c225d58-istiod-ca-cert\") pod \"1b335023-b1b8-4e94-888d-f8477c225d58\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") "
Sep 16 21:39:33.844790 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.844517    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-credential-socket\") pod \"1b335023-b1b8-4e94-888d-f8477c225d58\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") "
Sep 16 21:39:33.844790 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.844588    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-workload-socket\") pod \"1b335023-b1b8-4e94-888d-f8477c225d58\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") "
Sep 16 21:39:33.844790 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.844617    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b335023-b1b8-4e94-888d-f8477c225d58-istio-podinfo\") pod \"1b335023-b1b8-4e94-888d-f8477c225d58\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") "
Sep 16 21:39:33.844790 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.844648    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-istio-envoy\") pod \"1b335023-b1b8-4e94-888d-f8477c225d58\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") "
Sep 16 21:39:33.844790 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.844729    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-workload-certs\") pod \"1b335023-b1b8-4e94-888d-f8477c225d58\" (UID: \"1b335023-b1b8-4e94-888d-f8477c225d58\") "
Sep 16 21:39:33.845095 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.844782    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-istio-data" (OuterVolumeSpecName: "istio-data") pod "1b335023-b1b8-4e94-888d-f8477c225d58" (UID: "1b335023-b1b8-4e94-888d-f8477c225d58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:33.845095 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.844944    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1b335023-b1b8-4e94-888d-f8477c225d58" (UID: "1b335023-b1b8-4e94-888d-f8477c225d58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:33.845095 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.844956    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1b335023-b1b8-4e94-888d-f8477c225d58" (UID: "1b335023-b1b8-4e94-888d-f8477c225d58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:33.845095 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.845039    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b335023-b1b8-4e94-888d-f8477c225d58-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1b335023-b1b8-4e94-888d-f8477c225d58" (UID: "1b335023-b1b8-4e94-888d-f8477c225d58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:39:33.845296 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.845143    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1b335023-b1b8-4e94-888d-f8477c225d58" (UID: "1b335023-b1b8-4e94-888d-f8477c225d58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:33.845296 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.845271    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:33.845296 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.845292    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:33.845455 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.845306    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:33.845455 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.845319    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b335023-b1b8-4e94-888d-f8477c225d58-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:33.845455 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.845332    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:33.846904 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.846883    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1b335023-b1b8-4e94-888d-f8477c225d58" (UID: "1b335023-b1b8-4e94-888d-f8477c225d58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:33.846996 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.846975    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b335023-b1b8-4e94-888d-f8477c225d58-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1b335023-b1b8-4e94-888d-f8477c225d58" (UID: "1b335023-b1b8-4e94-888d-f8477c225d58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:39:33.847214 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.847192    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b335023-b1b8-4e94-888d-f8477c225d58-istio-token" (OuterVolumeSpecName: "istio-token") pod "1b335023-b1b8-4e94-888d-f8477c225d58" (UID: "1b335023-b1b8-4e94-888d-f8477c225d58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:39:33.847315 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.847220    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b335023-b1b8-4e94-888d-f8477c225d58-kube-api-access-2984g" (OuterVolumeSpecName: "kube-api-access-2984g") pod "1b335023-b1b8-4e94-888d-f8477c225d58" (UID: "1b335023-b1b8-4e94-888d-f8477c225d58"). InnerVolumeSpecName "kube-api-access-2984g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:39:33.946044 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.945996    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b335023-b1b8-4e94-888d-f8477c225d58-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:33.946044 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.946041    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b335023-b1b8-4e94-888d-f8477c225d58-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:33.946044 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.946050    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2984g\" (UniqueName: \"kubernetes.io/projected/1b335023-b1b8-4e94-888d-f8477c225d58-kube-api-access-2984g\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:33.946044 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:33.946060    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b335023-b1b8-4e94-888d-f8477c225d58-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:34.666787 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:34.666759    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"
Sep 16 21:39:34.667219 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:34.666756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl" event={"ID":"1b335023-b1b8-4e94-888d-f8477c225d58","Type":"ContainerDied","Data":"079b4cd4be43fb300fe58cef817c0f4de3a4d503ae99d17f806e3d30453bf430"}
Sep 16 21:39:34.667219 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:34.666903    2563 scope.go:117] "RemoveContainer" containerID="275d6c32183476b309edae7bb06d326a0841594700e8dadc0de78bd8dbd3d57c"
Sep 16 21:39:34.683508 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:34.683485    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"]
Sep 16 21:39:34.687388 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:34.687367    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-06ac85db-openshift-default-6df8fb649b-6x7tl"]
Sep 16 21:39:36.608639 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:36.608598    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b335023-b1b8-4e94-888d-f8477c225d58" path="/var/lib/kubelet/pods/1b335023-b1b8-4e94-888d-f8477c225d58/volumes"
Sep 16 21:39:39.861303 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:39.861268    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"]
Sep 16 21:39:41.809824 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:41.809785    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"]
Sep 16 21:39:41.810242 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:41.810061    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm" podUID="81f62c3e-ce8a-4860-acfb-d50aff0821cb" containerName="istio-proxy" containerID="cri-o://75ffef74499ddc1d0c7f3c8fc2967da552ae9cc1dbb3eb2ceebc8f758ecae6a7" gracePeriod=30
Sep 16 21:39:42.625983 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.625943    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"]
Sep 16 21:39:42.626475 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.626451    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b335023-b1b8-4e94-888d-f8477c225d58" containerName="istio-proxy"
Sep 16 21:39:42.626475 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.626472    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b335023-b1b8-4e94-888d-f8477c225d58" containerName="istio-proxy"
Sep 16 21:39:42.626715 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.626505    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="595f9075-eaa3-4bb6-9374-8c1ac173ed8f" containerName="revoke-keys"
Sep 16 21:39:42.626715 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.626511    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="595f9075-eaa3-4bb6-9374-8c1ac173ed8f" containerName="revoke-keys"
Sep 16 21:39:42.626715 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.626583    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b335023-b1b8-4e94-888d-f8477c225d58" containerName="istio-proxy"
Sep 16 21:39:42.626715 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.626592    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="595f9075-eaa3-4bb6-9374-8c1ac173ed8f" containerName="revoke-keys"
Sep 16 21:39:42.626715 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.626600    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b335023-b1b8-4e94-888d-f8477c225d58" containerName="istio-proxy"
Sep 16 21:39:42.631048 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.631031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.633226 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.633203    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-296002-openshift-default-dockercfg-ngwms\""
Sep 16 21:39:42.644394 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.644360    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"]
Sep 16 21:39:42.738173 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.738131    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-workload-certs\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.738430 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.738411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-podinfo\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.738558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.738542    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istiod-ca-cert\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.738730 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.738712    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-workload-socket\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.738884 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.738870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gqgr\" (UniqueName: \"kubernetes.io/projected/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-kube-api-access-7gqgr\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.739001 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.738987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-credential-socket\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.739257 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.739233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-data\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.739498 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.739479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-envoy\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.739710 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.739693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-token\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.840610 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.840576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-envoy\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.841054 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.840638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-token\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.841054 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.840689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-workload-certs\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.841054 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.840711    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-podinfo\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.841054 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.840726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istiod-ca-cert\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.841054 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.840811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-workload-socket\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.841054 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.840873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gqgr\" (UniqueName: \"kubernetes.io/projected/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-kube-api-access-7gqgr\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.841054 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.840919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-credential-socket\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.841054 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.840969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-data\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.841438 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.841152    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-workload-certs\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.841487 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.841432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-data\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.841487 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.841473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-workload-socket\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.841554 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.841488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-credential-socket\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.841554 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.841541    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istiod-ca-cert\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.843587 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.843559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-envoy\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.843779 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.843762    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-podinfo\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.863807 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.863771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-token\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.863950 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.863828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gqgr\" (UniqueName: \"kubernetes.io/projected/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-kube-api-access-7gqgr\") pod \"e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") " pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:42.941919 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:42.941882    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:43.218274 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:43.218242    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"]
Sep 16 21:39:43.220346 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:39:43.220317    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7831d5_d49f_41fe_9e06_c3f8c1027b11.slice/crio-aef5fc155336486c8aae11650c5ade3936a555356008b68dff25c1689ef02405 WatchSource:0}: Error finding container aef5fc155336486c8aae11650c5ade3936a555356008b68dff25c1689ef02405: Status 404 returned error can't find the container with id aef5fc155336486c8aae11650c5ade3936a555356008b68dff25c1689ef02405
Sep 16 21:39:43.222519 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:43.222487    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:39:43.222592 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:43.222569    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:39:43.222629 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:43.222597    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:39:43.722026 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:43.721988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp" event={"ID":"3d7831d5-d49f-41fe-9e06-c3f8c1027b11","Type":"ContainerStarted","Data":"3816449b88cf097d55a65f821cca63c3da8b07985f8c8b158d9e81acee028d8b"}
Sep 16 21:39:43.722214 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:43.722071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp" event={"ID":"3d7831d5-d49f-41fe-9e06-c3f8c1027b11","Type":"ContainerStarted","Data":"aef5fc155336486c8aae11650c5ade3936a555356008b68dff25c1689ef02405"}
Sep 16 21:39:43.742672 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:43.742604    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp" podStartSLOduration=1.742589194 podStartE2EDuration="1.742589194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:39:43.741093717 +0000 UTC m=+1043.641303721" watchObservedRunningTime="2026-09-16 21:39:43.742589194 +0000 UTC m=+1043.642799298"
Sep 16 21:39:43.862690 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:43.862640    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"]
Sep 16 21:39:43.942195 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:43.942157    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:44.947838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:44.947808    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:45.733637 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:45.733603    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:45.734630 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:45.734611    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:39:47.065959 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.065931    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:39:47.200956 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.200926    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istiod-ca-cert\") pod \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") "
Sep 16 21:39:47.201143 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201026    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gbw5\" (UniqueName: \"kubernetes.io/projected/81f62c3e-ce8a-4860-acfb-d50aff0821cb-kube-api-access-7gbw5\") pod \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") "
Sep 16 21:39:47.201143 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201065    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-credential-socket\") pod \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") "
Sep 16 21:39:47.201143 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201110    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-data\") pod \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") "
Sep 16 21:39:47.201304 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201176    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-token\") pod \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") "
Sep 16 21:39:47.201304 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201206    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-envoy\") pod \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") "
Sep 16 21:39:47.201304 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201247    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-workload-socket\") pod \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") "
Sep 16 21:39:47.201467 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201311    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-workload-certs\") pod \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") "
Sep 16 21:39:47.201467 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201365    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-podinfo\") pod \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\" (UID: \"81f62c3e-ce8a-4860-acfb-d50aff0821cb\") "
Sep 16 21:39:47.201467 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201373    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81f62c3e-ce8a-4860-acfb-d50aff0821cb" (UID: "81f62c3e-ce8a-4860-acfb-d50aff0821cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:47.201467 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201389    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-data" (OuterVolumeSpecName: "istio-data") pod "81f62c3e-ce8a-4860-acfb-d50aff0821cb" (UID: "81f62c3e-ce8a-4860-acfb-d50aff0821cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:47.201467 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201445    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81f62c3e-ce8a-4860-acfb-d50aff0821cb" (UID: "81f62c3e-ce8a-4860-acfb-d50aff0821cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:39:47.201745 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201564    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81f62c3e-ce8a-4860-acfb-d50aff0821cb" (UID: "81f62c3e-ce8a-4860-acfb-d50aff0821cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:47.201745 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201611    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81f62c3e-ce8a-4860-acfb-d50aff0821cb" (UID: "81f62c3e-ce8a-4860-acfb-d50aff0821cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:47.201825 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201800    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:47.201825 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201819    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:47.201903 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201832    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:47.201903 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201847    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:47.201903 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.201860    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:47.203451 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.203424    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81f62c3e-ce8a-4860-acfb-d50aff0821cb" (UID: "81f62c3e-ce8a-4860-acfb-d50aff0821cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:47.203564 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.203512    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81f62c3e-ce8a-4860-acfb-d50aff0821cb" (UID: "81f62c3e-ce8a-4860-acfb-d50aff0821cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:39:47.203828 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.203805    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f62c3e-ce8a-4860-acfb-d50aff0821cb-kube-api-access-7gbw5" (OuterVolumeSpecName: "kube-api-access-7gbw5") pod "81f62c3e-ce8a-4860-acfb-d50aff0821cb" (UID: "81f62c3e-ce8a-4860-acfb-d50aff0821cb"). InnerVolumeSpecName "kube-api-access-7gbw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:39:47.203896 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.203826    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-token" (OuterVolumeSpecName: "istio-token") pod "81f62c3e-ce8a-4860-acfb-d50aff0821cb" (UID: "81f62c3e-ce8a-4860-acfb-d50aff0821cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:39:47.302861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.302821    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gbw5\" (UniqueName: \"kubernetes.io/projected/81f62c3e-ce8a-4860-acfb-d50aff0821cb-kube-api-access-7gbw5\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:47.302861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.302857    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:47.302861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.302866    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:47.303121 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.302874    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f62c3e-ce8a-4860-acfb-d50aff0821cb-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:47.746112 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.746073    2563 generic.go:356] "Generic (PLEG): container finished" podID="81f62c3e-ce8a-4860-acfb-d50aff0821cb" containerID="75ffef74499ddc1d0c7f3c8fc2967da552ae9cc1dbb3eb2ceebc8f758ecae6a7" exitCode=0
Sep 16 21:39:47.746285 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.746124    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"
Sep 16 21:39:47.746285 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.746154    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm" event={"ID":"81f62c3e-ce8a-4860-acfb-d50aff0821cb","Type":"ContainerDied","Data":"75ffef74499ddc1d0c7f3c8fc2967da552ae9cc1dbb3eb2ceebc8f758ecae6a7"}
Sep 16 21:39:47.746285 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.746209    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm" event={"ID":"81f62c3e-ce8a-4860-acfb-d50aff0821cb","Type":"ContainerDied","Data":"5fb29c1f7e2f1bdc074f74dd9fa55dbd3c5b0b0d35c456e2747f8e76f487add7"}
Sep 16 21:39:47.746285 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.746232    2563 scope.go:117] "RemoveContainer" containerID="75ffef74499ddc1d0c7f3c8fc2967da552ae9cc1dbb3eb2ceebc8f758ecae6a7"
Sep 16 21:39:47.755990 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.755970    2563 scope.go:117] "RemoveContainer" containerID="75ffef74499ddc1d0c7f3c8fc2967da552ae9cc1dbb3eb2ceebc8f758ecae6a7"
Sep 16 21:39:47.756272 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:39:47.756250    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ffef74499ddc1d0c7f3c8fc2967da552ae9cc1dbb3eb2ceebc8f758ecae6a7\": container with ID starting with 75ffef74499ddc1d0c7f3c8fc2967da552ae9cc1dbb3eb2ceebc8f758ecae6a7 not found: ID does not exist" containerID="75ffef74499ddc1d0c7f3c8fc2967da552ae9cc1dbb3eb2ceebc8f758ecae6a7"
Sep 16 21:39:47.756321 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.756281    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ffef74499ddc1d0c7f3c8fc2967da552ae9cc1dbb3eb2ceebc8f758ecae6a7"} err="failed to get container status \"75ffef74499ddc1d0c7f3c8fc2967da552ae9cc1dbb3eb2ceebc8f758ecae6a7\": rpc error: code = NotFound desc = could not find container \"75ffef74499ddc1d0c7f3c8fc2967da552ae9cc1dbb3eb2ceebc8f758ecae6a7\": container with ID starting with 75ffef74499ddc1d0c7f3c8fc2967da552ae9cc1dbb3eb2ceebc8f758ecae6a7 not found: ID does not exist"
Sep 16 21:39:47.767570 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.767531    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"]
Sep 16 21:39:47.769667 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:47.769631    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-0a22a7-openshift-default-7848467d94-k7qhm"]
Sep 16 21:39:48.609095 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:48.609060    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81f62c3e-ce8a-4860-acfb-d50aff0821cb" path="/var/lib/kubelet/pods/81f62c3e-ce8a-4860-acfb-d50aff0821cb/volumes"
Sep 16 21:39:54.880984 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:54.880938    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq" podUID="0905dbe8-baa8-4af3-9ef2-4d4f058f6c12" containerName="main" containerID="cri-o://9dfc2779973a49e3b28b6c24529c1325aba8c26d9fa0a1c36c27f8ae196c5910" gracePeriod=45
Sep 16 21:39:55.172319 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.172293    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:39:55.294509 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.294473    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-dshm\") pod \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") "
Sep 16 21:39:55.294727 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.294538    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-model-cache\") pod \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") "
Sep 16 21:39:55.294727 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.294591    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psltz\" (UniqueName: \"kubernetes.io/projected/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-kube-api-access-psltz\") pod \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") "
Sep 16 21:39:55.294727 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.294622    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-tls-certs\") pod \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") "
Sep 16 21:39:55.294908 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.294756    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-home\") pod \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") "
Sep 16 21:39:55.294908 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.294803    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-tmp-dir\") pod \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\" (UID: \"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12\") "
Sep 16 21:39:55.294908 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.294807    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-model-cache" (OuterVolumeSpecName: "model-cache") pod "0905dbe8-baa8-4af3-9ef2-4d4f058f6c12" (UID: "0905dbe8-baa8-4af3-9ef2-4d4f058f6c12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:55.295070 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.295040    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-home" (OuterVolumeSpecName: "home") pod "0905dbe8-baa8-4af3-9ef2-4d4f058f6c12" (UID: "0905dbe8-baa8-4af3-9ef2-4d4f058f6c12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:55.295118 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.295064    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0905dbe8-baa8-4af3-9ef2-4d4f058f6c12" (UID: "0905dbe8-baa8-4af3-9ef2-4d4f058f6c12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:55.295266 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.295247    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-home\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:55.295317 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.295271    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-tmp-dir\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:55.295317 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.295287    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-model-cache\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:55.296926 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.296900    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0905dbe8-baa8-4af3-9ef2-4d4f058f6c12" (UID: "0905dbe8-baa8-4af3-9ef2-4d4f058f6c12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:39:55.297029 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.296926    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-dshm" (OuterVolumeSpecName: "dshm") pod "0905dbe8-baa8-4af3-9ef2-4d4f058f6c12" (UID: "0905dbe8-baa8-4af3-9ef2-4d4f058f6c12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:55.297029 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.296935    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-kube-api-access-psltz" (OuterVolumeSpecName: "kube-api-access-psltz") pod "0905dbe8-baa8-4af3-9ef2-4d4f058f6c12" (UID: "0905dbe8-baa8-4af3-9ef2-4d4f058f6c12"). InnerVolumeSpecName "kube-api-access-psltz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:39:55.396169 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.396133    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psltz\" (UniqueName: \"kubernetes.io/projected/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-kube-api-access-psltz\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:55.396169 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.396165    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-tls-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:55.396169 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.396176    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12-dshm\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:55.790088 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.790053    2563 generic.go:356] "Generic (PLEG): container finished" podID="0905dbe8-baa8-4af3-9ef2-4d4f058f6c12" containerID="9dfc2779973a49e3b28b6c24529c1325aba8c26d9fa0a1c36c27f8ae196c5910" exitCode=0
Sep 16 21:39:55.790280 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.790095    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"
Sep 16 21:39:55.790280 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.790131    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq" event={"ID":"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12","Type":"ContainerDied","Data":"9dfc2779973a49e3b28b6c24529c1325aba8c26d9fa0a1c36c27f8ae196c5910"}
Sep 16 21:39:55.790280 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.790174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq" event={"ID":"0905dbe8-baa8-4af3-9ef2-4d4f058f6c12","Type":"ContainerDied","Data":"fb07be037e1f2ad5dc824df85af81c279865eba3d60077e0cf75be9dfb339065"}
Sep 16 21:39:55.790280 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.790193    2563 scope.go:117] "RemoveContainer" containerID="9dfc2779973a49e3b28b6c24529c1325aba8c26d9fa0a1c36c27f8ae196c5910"
Sep 16 21:39:55.800119 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.800100    2563 scope.go:117] "RemoveContainer" containerID="9dfc2779973a49e3b28b6c24529c1325aba8c26d9fa0a1c36c27f8ae196c5910"
Sep 16 21:39:55.800407 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:39:55.800389    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dfc2779973a49e3b28b6c24529c1325aba8c26d9fa0a1c36c27f8ae196c5910\": container with ID starting with 9dfc2779973a49e3b28b6c24529c1325aba8c26d9fa0a1c36c27f8ae196c5910 not found: ID does not exist" containerID="9dfc2779973a49e3b28b6c24529c1325aba8c26d9fa0a1c36c27f8ae196c5910"
Sep 16 21:39:55.800489 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.800416    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dfc2779973a49e3b28b6c24529c1325aba8c26d9fa0a1c36c27f8ae196c5910"} err="failed to get container status \"9dfc2779973a49e3b28b6c24529c1325aba8c26d9fa0a1c36c27f8ae196c5910\": rpc error: code = NotFound desc = could not find container \"9dfc2779973a49e3b28b6c24529c1325aba8c26d9fa0a1c36c27f8ae196c5910\": container with ID starting with 9dfc2779973a49e3b28b6c24529c1325aba8c26d9fa0a1c36c27f8ae196c5910 not found: ID does not exist"
Sep 16 21:39:55.810939 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.810915    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"]
Sep 16 21:39:55.812806 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:55.812786    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e5568d/test-model-e5568d-kserve-7c5884779f-6bxjq"]
Sep 16 21:39:56.608604 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:56.608575    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0905dbe8-baa8-4af3-9ef2-4d4f058f6c12" path="/var/lib/kubelet/pods/0905dbe8-baa8-4af3-9ef2-4d4f058f6c12/volumes"
Sep 16 21:39:58.882368 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:58.882310    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5" podUID="68bb534f-c799-4262-85fc-4b3ad900c1ca" containerName="main" containerID="cri-o://e428561b5c69b166bb292d32305e331abaabeb7d5174397b6438911184bfa892" gracePeriod=45
Sep 16 21:39:59.129122 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.129094    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:39:59.133100 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.133045    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-model-cache\") pod \"68bb534f-c799-4262-85fc-4b3ad900c1ca\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") "
Sep 16 21:39:59.133220 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.133098    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-dshm\") pod \"68bb534f-c799-4262-85fc-4b3ad900c1ca\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") "
Sep 16 21:39:59.133220 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.133161    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-tmp-dir\") pod \"68bb534f-c799-4262-85fc-4b3ad900c1ca\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") "
Sep 16 21:39:59.133336 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.133223    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-home\") pod \"68bb534f-c799-4262-85fc-4b3ad900c1ca\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") "
Sep 16 21:39:59.133336 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.133268    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bb534f-c799-4262-85fc-4b3ad900c1ca-tls-certs\") pod \"68bb534f-c799-4262-85fc-4b3ad900c1ca\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") "
Sep 16 21:39:59.133336 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.133295    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "68bb534f-c799-4262-85fc-4b3ad900c1ca" (UID: "68bb534f-c799-4262-85fc-4b3ad900c1ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:59.133336 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.133308    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr5gm\" (UniqueName: \"kubernetes.io/projected/68bb534f-c799-4262-85fc-4b3ad900c1ca-kube-api-access-fr5gm\") pod \"68bb534f-c799-4262-85fc-4b3ad900c1ca\" (UID: \"68bb534f-c799-4262-85fc-4b3ad900c1ca\") "
Sep 16 21:39:59.133549 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.133471    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68bb534f-c799-4262-85fc-4b3ad900c1ca" (UID: "68bb534f-c799-4262-85fc-4b3ad900c1ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:59.133549 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.133484    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-home" (OuterVolumeSpecName: "home") pod "68bb534f-c799-4262-85fc-4b3ad900c1ca" (UID: "68bb534f-c799-4262-85fc-4b3ad900c1ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:59.133805 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.133788    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-model-cache\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:59.133805 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.133807    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-tmp-dir\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:59.133923 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.133816    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-home\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:59.135362 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.135340    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-dshm" (OuterVolumeSpecName: "dshm") pod "68bb534f-c799-4262-85fc-4b3ad900c1ca" (UID: "68bb534f-c799-4262-85fc-4b3ad900c1ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:39:59.135468 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.135427    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68bb534f-c799-4262-85fc-4b3ad900c1ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68bb534f-c799-4262-85fc-4b3ad900c1ca" (UID: "68bb534f-c799-4262-85fc-4b3ad900c1ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:39:59.135468 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.135445    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bb534f-c799-4262-85fc-4b3ad900c1ca-kube-api-access-fr5gm" (OuterVolumeSpecName: "kube-api-access-fr5gm") pod "68bb534f-c799-4262-85fc-4b3ad900c1ca" (UID: "68bb534f-c799-4262-85fc-4b3ad900c1ca"). InnerVolumeSpecName "kube-api-access-fr5gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:39:59.235030 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.234995    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68bb534f-c799-4262-85fc-4b3ad900c1ca-tls-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:59.235030 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.235027    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr5gm\" (UniqueName: \"kubernetes.io/projected/68bb534f-c799-4262-85fc-4b3ad900c1ca-kube-api-access-fr5gm\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:59.235030 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.235036    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68bb534f-c799-4262-85fc-4b3ad900c1ca-dshm\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:39:59.817454 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.817416    2563 generic.go:356] "Generic (PLEG): container finished" podID="68bb534f-c799-4262-85fc-4b3ad900c1ca" containerID="e428561b5c69b166bb292d32305e331abaabeb7d5174397b6438911184bfa892" exitCode=0
Sep 16 21:39:59.817694 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.817462    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"
Sep 16 21:39:59.817694 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.817505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5" event={"ID":"68bb534f-c799-4262-85fc-4b3ad900c1ca","Type":"ContainerDied","Data":"e428561b5c69b166bb292d32305e331abaabeb7d5174397b6438911184bfa892"}
Sep 16 21:39:59.817694 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.817552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5" event={"ID":"68bb534f-c799-4262-85fc-4b3ad900c1ca","Type":"ContainerDied","Data":"41c2b16768374c238b95cade3e665cb6f09442bce8fc2fcd7fed820ba22725cf"}
Sep 16 21:39:59.817694 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.817574    2563 scope.go:117] "RemoveContainer" containerID="e428561b5c69b166bb292d32305e331abaabeb7d5174397b6438911184bfa892"
Sep 16 21:39:59.827814 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.827793    2563 scope.go:117] "RemoveContainer" containerID="e428561b5c69b166bb292d32305e331abaabeb7d5174397b6438911184bfa892"
Sep 16 21:39:59.828116 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:39:59.828097    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e428561b5c69b166bb292d32305e331abaabeb7d5174397b6438911184bfa892\": container with ID starting with e428561b5c69b166bb292d32305e331abaabeb7d5174397b6438911184bfa892 not found: ID does not exist" containerID="e428561b5c69b166bb292d32305e331abaabeb7d5174397b6438911184bfa892"
Sep 16 21:39:59.828194 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.828123    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e428561b5c69b166bb292d32305e331abaabeb7d5174397b6438911184bfa892"} err="failed to get container status \"e428561b5c69b166bb292d32305e331abaabeb7d5174397b6438911184bfa892\": rpc error: code = NotFound desc = could not find container \"e428561b5c69b166bb292d32305e331abaabeb7d5174397b6438911184bfa892\": container with ID starting with e428561b5c69b166bb292d32305e331abaabeb7d5174397b6438911184bfa892 not found: ID does not exist"
Sep 16 21:39:59.849420 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.849387    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"]
Sep 16 21:39:59.854286 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:39:59.854262    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-44e517/test-model-44e517-kserve-c5db6b68f-hq7g5"]
Sep 16 21:40:00.608160 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:00.608106    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68bb534f-c799-4262-85fc-4b3ad900c1ca" path="/var/lib/kubelet/pods/68bb534f-c799-4262-85fc-4b3ad900c1ca/volumes"
Sep 16 21:40:02.067202 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.067165    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"]
Sep 16 21:40:02.067682 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.067588    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr" podUID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerName="istio-proxy" containerID="cri-o://226ecd91c09196b414f3ce34c95c835a5f788617d40dfcbad19d70eaedf790f3" gracePeriod=30
Sep 16 21:40:02.411773 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.411678    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm"]
Sep 16 21:40:02.412090 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.412076    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b335023-b1b8-4e94-888d-f8477c225d58" containerName="istio-proxy"
Sep 16 21:40:02.412128 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.412092    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b335023-b1b8-4e94-888d-f8477c225d58" containerName="istio-proxy"
Sep 16 21:40:02.412128 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.412116    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0905dbe8-baa8-4af3-9ef2-4d4f058f6c12" containerName="main"
Sep 16 21:40:02.412128 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.412121    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0905dbe8-baa8-4af3-9ef2-4d4f058f6c12" containerName="main"
Sep 16 21:40:02.412224 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.412131    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68bb534f-c799-4262-85fc-4b3ad900c1ca" containerName="main"
Sep 16 21:40:02.412224 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.412137    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68bb534f-c799-4262-85fc-4b3ad900c1ca" containerName="main"
Sep 16 21:40:02.412224 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.412145    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f62c3e-ce8a-4860-acfb-d50aff0821cb" containerName="istio-proxy"
Sep 16 21:40:02.412224 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.412153    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f62c3e-ce8a-4860-acfb-d50aff0821cb" containerName="istio-proxy"
Sep 16 21:40:02.412224 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.412217    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="81f62c3e-ce8a-4860-acfb-d50aff0821cb" containerName="istio-proxy"
Sep 16 21:40:02.412372 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.412229    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0905dbe8-baa8-4af3-9ef2-4d4f058f6c12" containerName="main"
Sep 16 21:40:02.412372 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.412239    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="68bb534f-c799-4262-85fc-4b3ad900c1ca" containerName="main"
Sep 16 21:40:02.416600 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.416579    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm"
Sep 16 21:40:02.418973 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.418946    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:40:02.419167 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.418981    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:40:02.421945 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.421922    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm"]
Sep 16 21:40:02.475986 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.475948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d4e070-dd26-4840-b8f3-8d086be52e01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm\" (UID: \"a2d4e070-dd26-4840-b8f3-8d086be52e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm"
Sep 16 21:40:02.577269 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.577235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d4e070-dd26-4840-b8f3-8d086be52e01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm\" (UID: \"a2d4e070-dd26-4840-b8f3-8d086be52e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm"
Sep 16 21:40:02.577925 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.577902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d4e070-dd26-4840-b8f3-8d086be52e01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm\" (UID: \"a2d4e070-dd26-4840-b8f3-8d086be52e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm"
Sep 16 21:40:02.729009 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:02.728971    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm"
Sep 16 21:40:03.008301 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:03.008276    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm"]
Sep 16 21:40:03.010276 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:40:03.010240    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d4e070_dd26_4840_b8f3_8d086be52e01.slice/crio-ed5573d614bb05c819d8fe709a4642f247a593374d2d58c6b26f6dcd4f151987 WatchSource:0}: Error finding container ed5573d614bb05c819d8fe709a4642f247a593374d2d58c6b26f6dcd4f151987: Status 404 returned error can't find the container with id ed5573d614bb05c819d8fe709a4642f247a593374d2d58c6b26f6dcd4f151987
Sep 16 21:40:03.842705 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:03.842648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm" event={"ID":"a2d4e070-dd26-4840-b8f3-8d086be52e01","Type":"ContainerStarted","Data":"4c762f004b45498832927e36fd21ea761e7b59da5f21bad2055f42eb503564e7"}
Sep 16 21:40:03.843144 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:03.842722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm" event={"ID":"a2d4e070-dd26-4840-b8f3-8d086be52e01","Type":"ContainerStarted","Data":"ed5573d614bb05c819d8fe709a4642f247a593374d2d58c6b26f6dcd4f151987"}
Sep 16 21:40:03.857877 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:03.857829    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm" podStartSLOduration=1.857814622 podStartE2EDuration="1.857814622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:03.856518853 +0000 UTC m=+1063.756728856" watchObservedRunningTime="2026-09-16 21:40:03.857814622 +0000 UTC m=+1063.758024625"
Sep 16 21:40:04.856075 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:04.856040    2563 generic.go:356] "Generic (PLEG): container finished" podID="a2d4e070-dd26-4840-b8f3-8d086be52e01" containerID="4c762f004b45498832927e36fd21ea761e7b59da5f21bad2055f42eb503564e7" exitCode=0
Sep 16 21:40:04.856468 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:04.856120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm" event={"ID":"a2d4e070-dd26-4840-b8f3-8d086be52e01","Type":"ContainerDied","Data":"4c762f004b45498832927e36fd21ea761e7b59da5f21bad2055f42eb503564e7"}
Sep 16 21:40:06.037273 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:06.037245    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm"
Sep 16 21:40:06.114411 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:06.114316    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d4e070-dd26-4840-b8f3-8d086be52e01-maas-api-service-ca\") pod \"a2d4e070-dd26-4840-b8f3-8d086be52e01\" (UID: \"a2d4e070-dd26-4840-b8f3-8d086be52e01\") "
Sep 16 21:40:06.114723 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:06.114698    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d4e070-dd26-4840-b8f3-8d086be52e01-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a2d4e070-dd26-4840-b8f3-8d086be52e01" (UID: "a2d4e070-dd26-4840-b8f3-8d086be52e01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:40:06.215498 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:06.215456    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d4e070-dd26-4840-b8f3-8d086be52e01-maas-api-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:06.870035 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:06.869939    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm"
Sep 16 21:40:06.870035 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:06.870020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm" event={"ID":"a2d4e070-dd26-4840-b8f3-8d086be52e01","Type":"ContainerDied","Data":"ed5573d614bb05c819d8fe709a4642f247a593374d2d58c6b26f6dcd4f151987"}
Sep 16 21:40:06.870284 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:06.870072    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed5573d614bb05c819d8fe709a4642f247a593374d2d58c6b26f6dcd4f151987"
Sep 16 21:40:07.348540 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.348515    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:40:07.429121 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429091    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-workload-certs\") pod \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") "
Sep 16 21:40:07.429300 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429128    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istiod-ca-cert\") pod \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") "
Sep 16 21:40:07.429300 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429154    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-podinfo\") pod \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") "
Sep 16 21:40:07.429300 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429193    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-envoy\") pod \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") "
Sep 16 21:40:07.429300 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429222    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-data\") pod \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") "
Sep 16 21:40:07.429300 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429285    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-workload-socket\") pod \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") "
Sep 16 21:40:07.429532 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429308    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s4vh\" (UniqueName: \"kubernetes.io/projected/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-kube-api-access-4s4vh\") pod \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") "
Sep 16 21:40:07.429532 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429330    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-token\") pod \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") "
Sep 16 21:40:07.429532 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429355    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" (UID: "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:40:07.429532 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429381    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-credential-socket\") pod \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\" (UID: \"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8\") "
Sep 16 21:40:07.429782 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429557    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" (UID: "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:40:07.429782 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429632    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-data" (OuterVolumeSpecName: "istio-data") pod "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" (UID: "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:40:07.429874 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429783    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:07.429874 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429783    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" (UID: "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:40:07.429874 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429801    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:07.429874 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429821    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:07.429874 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.429831    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" (UID: "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:40:07.431641 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.431612    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-kube-api-access-4s4vh" (OuterVolumeSpecName: "kube-api-access-4s4vh") pod "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" (UID: "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8"). InnerVolumeSpecName "kube-api-access-4s4vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:40:07.431778 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.431720    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" (UID: "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:40:07.431876 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.431857    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" (UID: "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:40:07.431979 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.431965    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-token" (OuterVolumeSpecName: "istio-token") pod "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" (UID: "aa4b1a1b-eaa1-428c-b315-ca46e5deafa8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:40:07.530601 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.530509    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:07.530601 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.530541    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:07.530601 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.530553    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:07.530601 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.530561    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:07.530601 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.530569    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:07.530601 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.530578    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s4vh\" (UniqueName: \"kubernetes.io/projected/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8-kube-api-access-4s4vh\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:07.880824 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.880725    2563 generic.go:356] "Generic (PLEG): container finished" podID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerID="226ecd91c09196b414f3ce34c95c835a5f788617d40dfcbad19d70eaedf790f3" exitCode=0
Sep 16 21:40:07.881007 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.880927    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"
Sep 16 21:40:07.891052 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.890760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr" event={"ID":"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8","Type":"ContainerDied","Data":"226ecd91c09196b414f3ce34c95c835a5f788617d40dfcbad19d70eaedf790f3"}
Sep 16 21:40:07.891052 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.890849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr" event={"ID":"aa4b1a1b-eaa1-428c-b315-ca46e5deafa8","Type":"ContainerDied","Data":"0af75074ac7710d4d3a18e8c50498939608676bb1ec093a195e2531e2689fec0"}
Sep 16 21:40:07.891052 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.890883    2563 scope.go:117] "RemoveContainer" containerID="226ecd91c09196b414f3ce34c95c835a5f788617d40dfcbad19d70eaedf790f3"
Sep 16 21:40:07.901646 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.901620    2563 scope.go:117] "RemoveContainer" containerID="c2095f774fd1e1360787e43484333f46f8b507dc8a6bf21dca4195453c28ae0d"
Sep 16 21:40:07.910195 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.910171    2563 scope.go:117] "RemoveContainer" containerID="226ecd91c09196b414f3ce34c95c835a5f788617d40dfcbad19d70eaedf790f3"
Sep 16 21:40:07.910497 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:07.910475    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226ecd91c09196b414f3ce34c95c835a5f788617d40dfcbad19d70eaedf790f3\": container with ID starting with 226ecd91c09196b414f3ce34c95c835a5f788617d40dfcbad19d70eaedf790f3 not found: ID does not exist" containerID="226ecd91c09196b414f3ce34c95c835a5f788617d40dfcbad19d70eaedf790f3"
Sep 16 21:40:07.910546 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.910507    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226ecd91c09196b414f3ce34c95c835a5f788617d40dfcbad19d70eaedf790f3"} err="failed to get container status \"226ecd91c09196b414f3ce34c95c835a5f788617d40dfcbad19d70eaedf790f3\": rpc error: code = NotFound desc = could not find container \"226ecd91c09196b414f3ce34c95c835a5f788617d40dfcbad19d70eaedf790f3\": container with ID starting with 226ecd91c09196b414f3ce34c95c835a5f788617d40dfcbad19d70eaedf790f3 not found: ID does not exist"
Sep 16 21:40:07.910546 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.910525    2563 scope.go:117] "RemoveContainer" containerID="c2095f774fd1e1360787e43484333f46f8b507dc8a6bf21dca4195453c28ae0d"
Sep 16 21:40:07.910809 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:07.910790    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2095f774fd1e1360787e43484333f46f8b507dc8a6bf21dca4195453c28ae0d\": container with ID starting with c2095f774fd1e1360787e43484333f46f8b507dc8a6bf21dca4195453c28ae0d not found: ID does not exist" containerID="c2095f774fd1e1360787e43484333f46f8b507dc8a6bf21dca4195453c28ae0d"
Sep 16 21:40:07.910882 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.910818    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2095f774fd1e1360787e43484333f46f8b507dc8a6bf21dca4195453c28ae0d"} err="failed to get container status \"c2095f774fd1e1360787e43484333f46f8b507dc8a6bf21dca4195453c28ae0d\": rpc error: code = NotFound desc = could not find container \"c2095f774fd1e1360787e43484333f46f8b507dc8a6bf21dca4195453c28ae0d\": container with ID starting with c2095f774fd1e1360787e43484333f46f8b507dc8a6bf21dca4195453c28ae0d not found: ID does not exist"
Sep 16 21:40:07.924806 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.924776    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"]
Sep 16 21:40:07.931927 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:07.931904    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e5568d-openshift-default-589747df4d-88vwr"]
Sep 16 21:40:08.608855 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:08.608825    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" path="/var/lib/kubelet/pods/aa4b1a1b-eaa1-428c-b315-ca46e5deafa8/volumes"
Sep 16 21:40:11.515001 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.514961    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"]
Sep 16 21:40:11.515513 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.515492    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d4e070-dd26-4840-b8f3-8d086be52e01" containerName="revoke-keys"
Sep 16 21:40:11.515513 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.515508    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d4e070-dd26-4840-b8f3-8d086be52e01" containerName="revoke-keys"
Sep 16 21:40:11.515611 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.515533    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerName="istio-proxy"
Sep 16 21:40:11.515611 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.515542    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerName="istio-proxy"
Sep 16 21:40:11.515611 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.515562    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerName="istio-proxy"
Sep 16 21:40:11.515611 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.515570    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerName="istio-proxy"
Sep 16 21:40:11.515769 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.515674    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerName="istio-proxy"
Sep 16 21:40:11.515769 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.515687    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa4b1a1b-eaa1-428c-b315-ca46e5deafa8" containerName="istio-proxy"
Sep 16 21:40:11.515769 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.515701    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2d4e070-dd26-4840-b8f3-8d086be52e01" containerName="revoke-keys"
Sep 16 21:40:11.519033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.519010    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.522012 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.521988    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-365ea1-openshift-default-dockercfg-ffjxs\""
Sep 16 21:40:11.530753 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.530714    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"]
Sep 16 21:40:11.575551 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.575453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-istio-data\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.575551 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.575512    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6553a042-53fe-40f5-843c-4815273f5737-istio-token\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.575838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.575589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-workload-socket\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.575838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.575705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-workload-certs\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.575838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.575740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6553a042-53fe-40f5-843c-4815273f5737-istiod-ca-cert\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.575838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.575769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-istio-envoy\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.575838 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.575806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-credential-socket\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.576057 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.575870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sh57\" (UniqueName: \"kubernetes.io/projected/6553a042-53fe-40f5-843c-4815273f5737-kube-api-access-2sh57\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.576057 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.575912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6553a042-53fe-40f5-843c-4815273f5737-istio-podinfo\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.676628 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.676581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-workload-certs\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.676628 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.676639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6553a042-53fe-40f5-843c-4815273f5737-istiod-ca-cert\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.676902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.676681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-istio-envoy\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.676902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.676706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-credential-socket\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.676902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.676730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sh57\" (UniqueName: \"kubernetes.io/projected/6553a042-53fe-40f5-843c-4815273f5737-kube-api-access-2sh57\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.676902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.676758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6553a042-53fe-40f5-843c-4815273f5737-istio-podinfo\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.677116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.676911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-istio-data\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.677116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.676972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6553a042-53fe-40f5-843c-4815273f5737-istio-token\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.677116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.677033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-workload-socket\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.677116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.677096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-workload-certs\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.677323 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.677168    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-credential-socket\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.677381 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.677357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-workload-socket\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.677381 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.677353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-istio-data\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.677501 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.677465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6553a042-53fe-40f5-843c-4815273f5737-istiod-ca-cert\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.679111 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.679088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-istio-envoy\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.679378 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.679357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6553a042-53fe-40f5-843c-4815273f5737-istio-podinfo\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.686352 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.686318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6553a042-53fe-40f5-843c-4815273f5737-istio-token\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.686452 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.686415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sh57\" (UniqueName: \"kubernetes.io/projected/6553a042-53fe-40f5-843c-4815273f5737-kube-api-access-2sh57\") pod \"e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") " pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.831300 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.831262    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:11.907289 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.907230    2563 generic.go:356] "Generic (PLEG): container finished" podID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" containerID="3816449b88cf097d55a65f821cca63c3da8b07985f8c8b158d9e81acee028d8b" exitCode=0
Sep 16 21:40:11.907456 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.907304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp" event={"ID":"3d7831d5-d49f-41fe-9e06-c3f8c1027b11","Type":"ContainerDied","Data":"3816449b88cf097d55a65f821cca63c3da8b07985f8c8b158d9e81acee028d8b"}
Sep 16 21:40:11.907941 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.907922    2563 scope.go:117] "RemoveContainer" containerID="3816449b88cf097d55a65f821cca63c3da8b07985f8c8b158d9e81acee028d8b"
Sep 16 21:40:11.909425 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.909273    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:11.909425 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.909387    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:11.909575 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:11.909449    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:12.110000 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:12.109803    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"]
Sep 16 21:40:12.112340 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:40:12.112305    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6553a042_53fe_40f5_843c_4815273f5737.slice/crio-09e106c121ceafaa53fc2ef5de87fd38e44dbc91f6229dc365c9c3c0651d7dba WatchSource:0}: Error finding container 09e106c121ceafaa53fc2ef5de87fd38e44dbc91f6229dc365c9c3c0651d7dba: Status 404 returned error can't find the container with id 09e106c121ceafaa53fc2ef5de87fd38e44dbc91f6229dc365c9c3c0651d7dba
Sep 16 21:40:12.115389 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:12.115347    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:12.115465 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:12.115444    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:12.115531 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:12.115512    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:12.914900 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:12.914861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl" event={"ID":"6553a042-53fe-40f5-843c-4815273f5737","Type":"ContainerStarted","Data":"d221954fca1b9ae36ffd19633c5fbfbc029df58cf6f3f4998ffa31580937f35f"}
Sep 16 21:40:12.915380 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:12.914921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl" event={"ID":"6553a042-53fe-40f5-843c-4815273f5737","Type":"ContainerStarted","Data":"09e106c121ceafaa53fc2ef5de87fd38e44dbc91f6229dc365c9c3c0651d7dba"}
Sep 16 21:40:12.916989 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:12.916969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp" event={"ID":"3d7831d5-d49f-41fe-9e06-c3f8c1027b11","Type":"ContainerStarted","Data":"60d64d5eecfc93d3be8faf49e75ac30643c84c1a033af46520d09fd4de5e750f"}
Sep 16 21:40:12.934173 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:12.934118    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl" podStartSLOduration=1.934102129 podStartE2EDuration="1.934102129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:12.932993438 +0000 UTC m=+1072.833203443" watchObservedRunningTime="2026-09-16 21:40:12.934102129 +0000 UTC m=+1072.834312132"
Sep 16 21:40:12.942034 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:12.942006    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:40:12.943319 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:12.943294    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 16 21:40:12.943432 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:12.943346    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp" podUID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 16 21:40:13.454611 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:13.454519    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm"]
Sep 16 21:40:13.457942 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:13.457910    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44e517-hx8wm"]
Sep 16 21:40:13.832171 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:13.832084    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:13.837012 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:13.836988    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:13.922795 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:13.922765    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:13.923691 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:13.923670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:13.942435 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:13.942406    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 16 21:40:13.942588 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:13.942504    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp" podUID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 16 21:40:14.608765 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:14.608724    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2d4e070-dd26-4840-b8f3-8d086be52e01" path="/var/lib/kubelet/pods/a2d4e070-dd26-4840-b8f3-8d086be52e01/volumes"
Sep 16 21:40:14.942492 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:14.942459    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 16 21:40:14.942885 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:14.942520    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp" podUID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 16 21:40:15.733918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:15.733870    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:40:15.943067 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:15.943031    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 16 21:40:15.943442 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:15.943121    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp" podUID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 16 21:40:16.943087 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:16.943050    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 16 21:40:16.943481 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:16.943106    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp" podUID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 16 21:40:17.942893 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:17.942861    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 16 21:40:17.943120 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:17.942929    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp" podUID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 16 21:40:18.947242 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:18.947204    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:40:18.948289 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:18.948268    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:40:21.134981 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:21.134945    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"]
Sep 16 21:40:21.135395 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:21.135192    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42" podUID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" containerName="istio-proxy" containerID="cri-o://16e1b5981f429adc87691fba1bab4d3ff5125d0ffb54d25e9d68adb0da3dcf11" gracePeriod=30
Sep 16 21:40:21.985940 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:21.985904    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"]
Sep 16 21:40:21.990824 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:21.990795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:21.992953 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:21.992930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-a7df81-openshift-default-dockercfg-h2wkh\""
Sep 16 21:40:22.005119 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.005092    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"]
Sep 16 21:40:22.092032 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.091417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-workload-socket\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.092032 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.091476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mftbb\" (UniqueName: \"kubernetes.io/projected/8fb9ef5a-b300-49de-8994-39e07ce01508-kube-api-access-mftbb\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.092032 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.091524    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-token\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.092032 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.091557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-workload-certs\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.092032 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.091605    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-podinfo\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.092032 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.091715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-credential-socket\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.092032 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.091778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-envoy\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.092032 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.091828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb9ef5a-b300-49de-8994-39e07ce01508-istiod-ca-cert\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.092032 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.091874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-data\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.192812 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.192774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-podinfo\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.193288 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.192842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-credential-socket\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.193288 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.192879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-envoy\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.193288 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.192910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb9ef5a-b300-49de-8994-39e07ce01508-istiod-ca-cert\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.193288 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.192936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-data\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.193288 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.192964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-workload-socket\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.193288 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.192987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mftbb\" (UniqueName: \"kubernetes.io/projected/8fb9ef5a-b300-49de-8994-39e07ce01508-kube-api-access-mftbb\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.193288 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.193026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-token\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.193288 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.193055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-workload-certs\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.193288 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.193258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-credential-socket\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.193787 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.193326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-workload-socket\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.193787 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.193368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-data\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.193787 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.193398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-workload-certs\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.194049 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.194027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb9ef5a-b300-49de-8994-39e07ce01508-istiod-ca-cert\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.195345 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.195319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-envoy\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.195580 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.195563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-podinfo\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.202241 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.202220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-token\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.202365 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.202261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mftbb\" (UniqueName: \"kubernetes.io/projected/8fb9ef5a-b300-49de-8994-39e07ce01508-kube-api-access-mftbb\") pod \"e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") " pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.303046 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.302954    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:22.581207 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.581179    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"]
Sep 16 21:40:22.582845 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:40:22.582818    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb9ef5a_b300_49de_8994_39e07ce01508.slice/crio-1210f57a5e90f017c10d849a814bc7d568770987f4189047da4ba4ec332495bd WatchSource:0}: Error finding container 1210f57a5e90f017c10d849a814bc7d568770987f4189047da4ba4ec332495bd: Status 404 returned error can't find the container with id 1210f57a5e90f017c10d849a814bc7d568770987f4189047da4ba4ec332495bd
Sep 16 21:40:22.585271 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.585238    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:22.585355 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.585304    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:22.585355 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.585331    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:22.975377 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.975338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t" event={"ID":"8fb9ef5a-b300-49de-8994-39e07ce01508","Type":"ContainerStarted","Data":"5f5d3140496b8c8119b4316029763adc3e2343b391104dac3f83b68fd73167c1"}
Sep 16 21:40:22.975529 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.975401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t" event={"ID":"8fb9ef5a-b300-49de-8994-39e07ce01508","Type":"ContainerStarted","Data":"1210f57a5e90f017c10d849a814bc7d568770987f4189047da4ba4ec332495bd"}
Sep 16 21:40:22.995032 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:22.994982    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t" podStartSLOduration=1.99496601 podStartE2EDuration="1.99496601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:22.993887977 +0000 UTC m=+1082.894097981" watchObservedRunningTime="2026-09-16 21:40:22.99496601 +0000 UTC m=+1082.895176014"
Sep 16 21:40:23.303864 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:23.303767    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:23.309106 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:23.309077    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:23.980599 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:23.980570    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:23.981517 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:23.981498    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:26.385359 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.385330    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:40:26.539498 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.539468    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istiod-ca-cert\") pod \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") "
Sep 16 21:40:26.539767 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.539561    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-data\") pod \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") "
Sep 16 21:40:26.539767 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.539589    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-workload-certs\") pod \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") "
Sep 16 21:40:26.539767 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.539611    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-workload-socket\") pod \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") "
Sep 16 21:40:26.539767 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.539650    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-credential-socket\") pod \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") "
Sep 16 21:40:26.539767 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.539688    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-envoy\") pod \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") "
Sep 16 21:40:26.539767 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.539717    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-token\") pod \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") "
Sep 16 21:40:26.539767 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.539750    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcz6z\" (UniqueName: \"kubernetes.io/projected/83813d1b-be84-4dbc-a6a3-8f8f2f850486-kube-api-access-lcz6z\") pod \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") "
Sep 16 21:40:26.540172 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.539796    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-podinfo\") pod \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\" (UID: \"83813d1b-be84-4dbc-a6a3-8f8f2f850486\") "
Sep 16 21:40:26.540172 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.539837    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-data" (OuterVolumeSpecName: "istio-data") pod "83813d1b-be84-4dbc-a6a3-8f8f2f850486" (UID: "83813d1b-be84-4dbc-a6a3-8f8f2f850486"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:40:26.540172 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.539881    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "83813d1b-be84-4dbc-a6a3-8f8f2f850486" (UID: "83813d1b-be84-4dbc-a6a3-8f8f2f850486"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:40:26.540172 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.539902    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "83813d1b-be84-4dbc-a6a3-8f8f2f850486" (UID: "83813d1b-be84-4dbc-a6a3-8f8f2f850486"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:40:26.540172 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.539921    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "83813d1b-be84-4dbc-a6a3-8f8f2f850486" (UID: "83813d1b-be84-4dbc-a6a3-8f8f2f850486"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:40:26.540172 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.540026    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "83813d1b-be84-4dbc-a6a3-8f8f2f850486" (UID: "83813d1b-be84-4dbc-a6a3-8f8f2f850486"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:40:26.540471 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.540248    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:26.540471 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.540267    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:26.540471 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.540280    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:26.540471 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.540292    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:26.540471 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.540304    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:26.542334 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.542307    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "83813d1b-be84-4dbc-a6a3-8f8f2f850486" (UID: "83813d1b-be84-4dbc-a6a3-8f8f2f850486"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:40:26.542334 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.542311    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "83813d1b-be84-4dbc-a6a3-8f8f2f850486" (UID: "83813d1b-be84-4dbc-a6a3-8f8f2f850486"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:40:26.542526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.542370    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83813d1b-be84-4dbc-a6a3-8f8f2f850486-kube-api-access-lcz6z" (OuterVolumeSpecName: "kube-api-access-lcz6z") pod "83813d1b-be84-4dbc-a6a3-8f8f2f850486" (UID: "83813d1b-be84-4dbc-a6a3-8f8f2f850486"). InnerVolumeSpecName "kube-api-access-lcz6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:40:26.542590 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.542530    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-token" (OuterVolumeSpecName: "istio-token") pod "83813d1b-be84-4dbc-a6a3-8f8f2f850486" (UID: "83813d1b-be84-4dbc-a6a3-8f8f2f850486"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:40:26.641489 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.641459    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:26.641489 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.641486    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:26.641489 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.641496    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcz6z\" (UniqueName: \"kubernetes.io/projected/83813d1b-be84-4dbc-a6a3-8f8f2f850486-kube-api-access-lcz6z\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:26.641741 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:26.641506    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83813d1b-be84-4dbc-a6a3-8f8f2f850486-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:27.004458 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:27.004419    2563 generic.go:356] "Generic (PLEG): container finished" podID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" containerID="16e1b5981f429adc87691fba1bab4d3ff5125d0ffb54d25e9d68adb0da3dcf11" exitCode=0
Sep 16 21:40:27.004871 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:27.004514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42" event={"ID":"83813d1b-be84-4dbc-a6a3-8f8f2f850486","Type":"ContainerDied","Data":"16e1b5981f429adc87691fba1bab4d3ff5125d0ffb54d25e9d68adb0da3dcf11"}
Sep 16 21:40:27.005108 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:27.005090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42" event={"ID":"83813d1b-be84-4dbc-a6a3-8f8f2f850486","Type":"ContainerDied","Data":"5b9be0a43f1f2c13df093dfa9b60221464124783758c030412cd3d452c56bb2e"}
Sep 16 21:40:27.005262 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:27.005213    2563 scope.go:117] "RemoveContainer" containerID="16e1b5981f429adc87691fba1bab4d3ff5125d0ffb54d25e9d68adb0da3dcf11"
Sep 16 21:40:27.005482 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:27.004522    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"
Sep 16 21:40:27.029187 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:27.029156    2563 scope.go:117] "RemoveContainer" containerID="96ea0f757e4b4fc9734b24c7d23f515689ad30afc253d29744c6e762f9c0d7e0"
Sep 16 21:40:27.032812 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:27.032734    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"]
Sep 16 21:40:27.035990 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:27.035899    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-44e517-openshift-default-59cf864b8-6ns42"]
Sep 16 21:40:27.042578 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:27.042499    2563 scope.go:117] "RemoveContainer" containerID="16e1b5981f429adc87691fba1bab4d3ff5125d0ffb54d25e9d68adb0da3dcf11"
Sep 16 21:40:27.042952 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:27.042922    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e1b5981f429adc87691fba1bab4d3ff5125d0ffb54d25e9d68adb0da3dcf11\": container with ID starting with 16e1b5981f429adc87691fba1bab4d3ff5125d0ffb54d25e9d68adb0da3dcf11 not found: ID does not exist" containerID="16e1b5981f429adc87691fba1bab4d3ff5125d0ffb54d25e9d68adb0da3dcf11"
Sep 16 21:40:27.043068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:27.042963    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e1b5981f429adc87691fba1bab4d3ff5125d0ffb54d25e9d68adb0da3dcf11"} err="failed to get container status \"16e1b5981f429adc87691fba1bab4d3ff5125d0ffb54d25e9d68adb0da3dcf11\": rpc error: code = NotFound desc = could not find container \"16e1b5981f429adc87691fba1bab4d3ff5125d0ffb54d25e9d68adb0da3dcf11\": container with ID starting with 16e1b5981f429adc87691fba1bab4d3ff5125d0ffb54d25e9d68adb0da3dcf11 not found: ID does not exist"
Sep 16 21:40:27.043068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:27.042989    2563 scope.go:117] "RemoveContainer" containerID="96ea0f757e4b4fc9734b24c7d23f515689ad30afc253d29744c6e762f9c0d7e0"
Sep 16 21:40:27.043584 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:27.043554    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ea0f757e4b4fc9734b24c7d23f515689ad30afc253d29744c6e762f9c0d7e0\": container with ID starting with 96ea0f757e4b4fc9734b24c7d23f515689ad30afc253d29744c6e762f9c0d7e0 not found: ID does not exist" containerID="96ea0f757e4b4fc9734b24c7d23f515689ad30afc253d29744c6e762f9c0d7e0"
Sep 16 21:40:27.043696 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:27.043588    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ea0f757e4b4fc9734b24c7d23f515689ad30afc253d29744c6e762f9c0d7e0"} err="failed to get container status \"96ea0f757e4b4fc9734b24c7d23f515689ad30afc253d29744c6e762f9c0d7e0\": rpc error: code = NotFound desc = could not find container \"96ea0f757e4b4fc9734b24c7d23f515689ad30afc253d29744c6e762f9c0d7e0\": container with ID starting with 96ea0f757e4b4fc9734b24c7d23f515689ad30afc253d29744c6e762f9c0d7e0 not found: ID does not exist"
Sep 16 21:40:28.607967 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:28.607932    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" path="/var/lib/kubelet/pods/83813d1b-be84-4dbc-a6a3-8f8f2f850486/volumes"
Sep 16 21:40:35.625233 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:35.625190    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7372-ec2834"]
Sep 16 21:40:35.625680 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:35.625614    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" containerName="istio-proxy"
Sep 16 21:40:35.625680 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:35.625626    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" containerName="istio-proxy"
Sep 16 21:40:35.625680 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:35.625633    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" containerName="istio-proxy"
Sep 16 21:40:35.625680 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:35.625638    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" containerName="istio-proxy"
Sep 16 21:40:35.625853 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:35.625713    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" containerName="istio-proxy"
Sep 16 21:40:35.630109 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:35.630090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-ec2834"
Sep 16 21:40:35.632615 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:35.632588    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8588c\""
Sep 16 21:40:35.635479 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:35.635452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7372-ec2834"]
Sep 16 21:40:35.645994 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:35.645953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9zkf\" (UniqueName: \"kubernetes.io/projected/2a7ef5ed-383c-424e-b456-ff6c8794313e-kube-api-access-m9zkf\") pod \"test-curl-7372-ec2834\" (UID: \"2a7ef5ed-383c-424e-b456-ff6c8794313e\") " pod="opendatahub/test-curl-7372-ec2834"
Sep 16 21:40:35.747174 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:35.747133    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9zkf\" (UniqueName: \"kubernetes.io/projected/2a7ef5ed-383c-424e-b456-ff6c8794313e-kube-api-access-m9zkf\") pod \"test-curl-7372-ec2834\" (UID: \"2a7ef5ed-383c-424e-b456-ff6c8794313e\") " pod="opendatahub/test-curl-7372-ec2834"
Sep 16 21:40:35.757487 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:35.757459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9zkf\" (UniqueName: \"kubernetes.io/projected/2a7ef5ed-383c-424e-b456-ff6c8794313e-kube-api-access-m9zkf\") pod \"test-curl-7372-ec2834\" (UID: \"2a7ef5ed-383c-424e-b456-ff6c8794313e\") " pod="opendatahub/test-curl-7372-ec2834"
Sep 16 21:40:35.942345 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:35.942306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-ec2834"
Sep 16 21:40:36.054005 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:36.053967    2563 generic.go:356] "Generic (PLEG): container finished" podID="6553a042-53fe-40f5-843c-4815273f5737" containerID="d221954fca1b9ae36ffd19633c5fbfbc029df58cf6f3f4998ffa31580937f35f" exitCode=0
Sep 16 21:40:36.054178 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:36.054011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl" event={"ID":"6553a042-53fe-40f5-843c-4815273f5737","Type":"ContainerDied","Data":"d221954fca1b9ae36ffd19633c5fbfbc029df58cf6f3f4998ffa31580937f35f"}
Sep 16 21:40:36.054417 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:36.054405    2563 scope.go:117] "RemoveContainer" containerID="d221954fca1b9ae36ffd19633c5fbfbc029df58cf6f3f4998ffa31580937f35f"
Sep 16 21:40:36.055612 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:36.055580    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:36.055730 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:36.055680    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:36.055730 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:36.055714    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:36.211953 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:36.211927    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7372-ec2834"]
Sep 16 21:40:36.214259 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:40:36.214231    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a7ef5ed_383c_424e_b456_ff6c8794313e.slice/crio-380ba6c80621b4a527014e1278185614de0e883f0989b807f46d33a814b7c5f2 WatchSource:0}: Error finding container 380ba6c80621b4a527014e1278185614de0e883f0989b807f46d33a814b7c5f2: Status 404 returned error can't find the container with id 380ba6c80621b4a527014e1278185614de0e883f0989b807f46d33a814b7c5f2
Sep 16 21:40:36.216290 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:36.216272    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 21:40:37.061949 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:37.061914    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl" event={"ID":"6553a042-53fe-40f5-843c-4815273f5737","Type":"ContainerStarted","Data":"0427ecb74d2eb742841649bb4749b202407ca907a8eab342740f491ac815d1e0"}
Sep 16 21:40:37.063718 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:37.063687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7372-ec2834" event={"ID":"2a7ef5ed-383c-424e-b456-ff6c8794313e","Type":"ContainerStarted","Data":"380ba6c80621b4a527014e1278185614de0e883f0989b807f46d33a814b7c5f2"}
Sep 16 21:40:37.831523 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:37.831470    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:37.831765 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:37.831737    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 21:40:37.831859 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:37.831803    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl" podUID="6553a042-53fe-40f5-843c-4815273f5737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 21:40:38.074760 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:38.074719    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7372-ec2834" event={"ID":"2a7ef5ed-383c-424e-b456-ff6c8794313e","Type":"ContainerStarted","Data":"0f63b8b03528cbf6f69acc36415549b16b2577ca7993bb35a8622a2f7e731b3e"}
Sep 16 21:40:38.090299 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:38.090145    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7372-ec2834" podStartSLOduration=2.319969978 podStartE2EDuration="3.090122374s" totalImagesPullingTime="770.152396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:38.090028811 +0000 UTC m=+1097.990238814" watchObservedRunningTime="2026-09-16 21:40:38.090122374 +0000 UTC m=+1097.990332379"
Sep 16 21:40:38.749609 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:38.749515    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7372-ec2834"]
Sep 16 21:40:38.753788 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:38.753759    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7372-ec2834"]
Sep 16 21:40:38.832526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:38.832485    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 21:40:38.832721 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:38.832559    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl" podUID="6553a042-53fe-40f5-843c-4815273f5737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 21:40:38.888677 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:38.888301    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7372-442938"]
Sep 16 21:40:38.889681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:38.888949    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7ef5ed-383c-424e-b456-ff6c8794313e" containerName="test-curl-7372-ec2834"
Sep 16 21:40:38.889681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:38.888974    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7ef5ed-383c-424e-b456-ff6c8794313e" containerName="test-curl-7372-ec2834"
Sep 16 21:40:38.889681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:38.889057    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7ef5ed-383c-424e-b456-ff6c8794313e" containerName="test-curl-7372-ec2834"
Sep 16 21:40:38.889681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:38.889073    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="83813d1b-be84-4dbc-a6a3-8f8f2f850486" containerName="istio-proxy"
Sep 16 21:40:38.892671 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:38.892636    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-442938"
Sep 16 21:40:38.906050 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:38.906018    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7372-442938"]
Sep 16 21:40:38.996057 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:38.996015    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxnb\" (UniqueName: \"kubernetes.io/projected/091a2527-3996-4e6e-8993-2c0a4e86d2e2-kube-api-access-rsxnb\") pod \"test-curl-7372-442938\" (UID: \"091a2527-3996-4e6e-8993-2c0a4e86d2e2\") " pod="opendatahub/test-curl-7372-442938"
Sep 16 21:40:39.082392 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:39.082286    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7372-ec2834" container="test-curl-7372-ec2834"
Sep 16 21:40:39.097702 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:39.097627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxnb\" (UniqueName: \"kubernetes.io/projected/091a2527-3996-4e6e-8993-2c0a4e86d2e2-kube-api-access-rsxnb\") pod \"test-curl-7372-442938\" (UID: \"091a2527-3996-4e6e-8993-2c0a4e86d2e2\") " pod="opendatahub/test-curl-7372-442938"
Sep 16 21:40:39.110212 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:39.110180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxnb\" (UniqueName: \"kubernetes.io/projected/091a2527-3996-4e6e-8993-2c0a4e86d2e2-kube-api-access-rsxnb\") pod \"test-curl-7372-442938\" (UID: \"091a2527-3996-4e6e-8993-2c0a4e86d2e2\") " pod="opendatahub/test-curl-7372-442938"
Sep 16 21:40:39.202158 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:39.202122    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-442938"
Sep 16 21:40:39.480491 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:39.480445    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7372-442938"]
Sep 16 21:40:39.492527 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:40:39.492493    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod091a2527_3996_4e6e_8993_2c0a4e86d2e2.slice/crio-01e43545f331acd2367df012ddcd438ef0abc2cc211ef5ef4e139987d9ac0198 WatchSource:0}: Error finding container 01e43545f331acd2367df012ddcd438ef0abc2cc211ef5ef4e139987d9ac0198: Status 404 returned error can't find the container with id 01e43545f331acd2367df012ddcd438ef0abc2cc211ef5ef4e139987d9ac0198
Sep 16 21:40:39.832493 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:39.832398    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 21:40:39.832493 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:39.832468    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl" podUID="6553a042-53fe-40f5-843c-4815273f5737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 21:40:40.092136 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:40.092030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7372-442938" event={"ID":"091a2527-3996-4e6e-8993-2c0a4e86d2e2","Type":"ContainerStarted","Data":"162fd83d24dbf3f15c922d781f5d1695587701028a98916f30e1a5dd84e1cac3"}
Sep 16 21:40:40.092136 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:40.092091    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7372-442938" event={"ID":"091a2527-3996-4e6e-8993-2c0a4e86d2e2","Type":"ContainerStarted","Data":"01e43545f331acd2367df012ddcd438ef0abc2cc211ef5ef4e139987d9ac0198"}
Sep 16 21:40:40.092678 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:40.092170    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7372-ec2834" podUID="2a7ef5ed-383c-424e-b456-ff6c8794313e" containerName="test-curl-7372-ec2834" containerID="cri-o://0f63b8b03528cbf6f69acc36415549b16b2577ca7993bb35a8622a2f7e731b3e" gracePeriod=2
Sep 16 21:40:40.127498 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:40.127440    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7372-442938" podStartSLOduration=2.127423311 podStartE2EDuration="2.127423311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:40.124708667 +0000 UTC m=+1100.024918672" watchObservedRunningTime="2026-09-16 21:40:40.127423311 +0000 UTC m=+1100.027633352"
Sep 16 21:40:40.127857 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:40.127827    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7372-ec2834\" is forbidden: User \"system:node:ip-10-0-136-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-118.ec2.internal' and this object" podUID="2a7ef5ed-383c-424e-b456-ff6c8794313e" pod="opendatahub/test-curl-7372-ec2834"
Sep 16 21:40:40.130199 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:40.130170    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7372-ec2834\" is forbidden: User \"system:node:ip-10-0-136-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-118.ec2.internal' and this object" podUID="2a7ef5ed-383c-424e-b456-ff6c8794313e" pod="opendatahub/test-curl-7372-ec2834"
Sep 16 21:40:40.606840 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:40.606801    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7372-ec2834\" is forbidden: User \"system:node:ip-10-0-136-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-118.ec2.internal' and this object" podUID="2a7ef5ed-383c-424e-b456-ff6c8794313e" pod="opendatahub/test-curl-7372-ec2834"
Sep 16 21:40:40.820600 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:40.820514    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7372-442938"]
Sep 16 21:40:40.823594 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:40.823568    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7372-442938"]
Sep 16 21:40:40.832120 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:40.832085    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 21:40:40.832235 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:40.832158    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl" podUID="6553a042-53fe-40f5-843c-4815273f5737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 21:40:40.946598 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:40.946562    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7372-b3db15"]
Sep 16 21:40:40.946959 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:40.946946    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="091a2527-3996-4e6e-8993-2c0a4e86d2e2" containerName="test-curl-7372-442938"
Sep 16 21:40:40.947006 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:40.946960    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="091a2527-3996-4e6e-8993-2c0a4e86d2e2" containerName="test-curl-7372-442938"
Sep 16 21:40:40.947048 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:40.947014    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="091a2527-3996-4e6e-8993-2c0a4e86d2e2" containerName="test-curl-7372-442938"
Sep 16 21:40:40.950120 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:40.950105    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-b3db15"
Sep 16 21:40:40.954826 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:40.954797    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7372-b3db15"]
Sep 16 21:40:41.016557 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:41.016518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd4v2\" (UniqueName: \"kubernetes.io/projected/060d1891-ce8c-4961-ae38-0f9657672574-kube-api-access-dd4v2\") pod \"test-curl-7372-b3db15\" (UID: \"060d1891-ce8c-4961-ae38-0f9657672574\") " pod="opendatahub/test-curl-7372-b3db15"
Sep 16 21:40:41.097356 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:41.097250    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7372-442938" container="test-curl-7372-442938"
Sep 16 21:40:41.117691 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:41.117621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd4v2\" (UniqueName: \"kubernetes.io/projected/060d1891-ce8c-4961-ae38-0f9657672574-kube-api-access-dd4v2\") pod \"test-curl-7372-b3db15\" (UID: \"060d1891-ce8c-4961-ae38-0f9657672574\") " pod="opendatahub/test-curl-7372-b3db15"
Sep 16 21:40:41.132430 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:41.132394    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd4v2\" (UniqueName: \"kubernetes.io/projected/060d1891-ce8c-4961-ae38-0f9657672574-kube-api-access-dd4v2\") pod \"test-curl-7372-b3db15\" (UID: \"060d1891-ce8c-4961-ae38-0f9657672574\") " pod="opendatahub/test-curl-7372-b3db15"
Sep 16 21:40:41.261430 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:41.261382    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-b3db15"
Sep 16 21:40:41.553429 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:41.553397    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7372-b3db15"]
Sep 16 21:40:41.555364 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:40:41.555338    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod060d1891_ce8c_4961_ae38_0f9657672574.slice/crio-12a131d6e6fce96207f96a75ac1be089a6e0c52b28356d6bfeff6ad0b4d30913 WatchSource:0}: Error finding container 12a131d6e6fce96207f96a75ac1be089a6e0c52b28356d6bfeff6ad0b4d30913: Status 404 returned error can't find the container with id 12a131d6e6fce96207f96a75ac1be089a6e0c52b28356d6bfeff6ad0b4d30913
Sep 16 21:40:41.832255 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:41.832160    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 21:40:41.832255 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:41.832219    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl" podUID="6553a042-53fe-40f5-843c-4815273f5737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 21:40:42.105848 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.105814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7372-b3db15" event={"ID":"060d1891-ce8c-4961-ae38-0f9657672574","Type":"ContainerStarted","Data":"25cbd64d0fbb97286f7a849e7e31a8ea11473b09b5367bd83e27b4131fc8d87f"}
Sep 16 21:40:42.106272 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.105859    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7372-b3db15" event={"ID":"060d1891-ce8c-4961-ae38-0f9657672574","Type":"ContainerStarted","Data":"12a131d6e6fce96207f96a75ac1be089a6e0c52b28356d6bfeff6ad0b4d30913"}
Sep 16 21:40:42.106272 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.105922    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7372-442938" podUID="091a2527-3996-4e6e-8993-2c0a4e86d2e2" containerName="test-curl-7372-442938" containerID="cri-o://162fd83d24dbf3f15c922d781f5d1695587701028a98916f30e1a5dd84e1cac3" gracePeriod=2
Sep 16 21:40:42.120627 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:42.120580    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7372-442938\" is forbidden: User \"system:node:ip-10-0-136-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-118.ec2.internal' and this object" podUID="091a2527-3996-4e6e-8993-2c0a4e86d2e2" pod="opendatahub/test-curl-7372-442938"
Sep 16 21:40:42.120894 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.120854    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7372-b3db15" podStartSLOduration=2.120838831 podStartE2EDuration="2.120838831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:42.118878913 +0000 UTC m=+1102.019088917" watchObservedRunningTime="2026-09-16 21:40:42.120838831 +0000 UTC m=+1102.021048833"
Sep 16 21:40:42.122103 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:42.122079    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7372-442938\" is forbidden: User \"system:node:ip-10-0-136-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-118.ec2.internal' and this object" podUID="091a2527-3996-4e6e-8993-2c0a4e86d2e2" pod="opendatahub/test-curl-7372-442938"
Sep 16 21:40:42.265481 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.265450    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-ec2834"
Sep 16 21:40:42.267591 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:42.267558    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7372-442938\" is forbidden: User \"system:node:ip-10-0-136-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-118.ec2.internal' and this object" podUID="091a2527-3996-4e6e-8993-2c0a4e86d2e2" pod="opendatahub/test-curl-7372-442938"
Sep 16 21:40:42.332129 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.332097    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9zkf\" (UniqueName: \"kubernetes.io/projected/2a7ef5ed-383c-424e-b456-ff6c8794313e-kube-api-access-m9zkf\") pod \"2a7ef5ed-383c-424e-b456-ff6c8794313e\" (UID: \"2a7ef5ed-383c-424e-b456-ff6c8794313e\") "
Sep 16 21:40:42.335247 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.335212    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7ef5ed-383c-424e-b456-ff6c8794313e-kube-api-access-m9zkf" (OuterVolumeSpecName: "kube-api-access-m9zkf") pod "2a7ef5ed-383c-424e-b456-ff6c8794313e" (UID: "2a7ef5ed-383c-424e-b456-ff6c8794313e"). InnerVolumeSpecName "kube-api-access-m9zkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:40:42.433513 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.433479    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9zkf\" (UniqueName: \"kubernetes.io/projected/2a7ef5ed-383c-424e-b456-ff6c8794313e-kube-api-access-m9zkf\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:42.609191 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.609159    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a7ef5ed-383c-424e-b456-ff6c8794313e" path="/var/lib/kubelet/pods/2a7ef5ed-383c-424e-b456-ff6c8794313e/volumes"
Sep 16 21:40:42.702758 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.702650    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7372-b3db15"]
Sep 16 21:40:42.706413 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.706384    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7372-b3db15"]
Sep 16 21:40:42.837104 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.837069    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:42.837360 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.837329    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7372-e7bdb1"]
Sep 16 21:40:42.837865 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.837843    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060d1891-ce8c-4961-ae38-0f9657672574" containerName="test-curl-7372-b3db15"
Sep 16 21:40:42.837957 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.837869    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="060d1891-ce8c-4961-ae38-0f9657672574" containerName="test-curl-7372-b3db15"
Sep 16 21:40:42.838002 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.837973    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="060d1891-ce8c-4961-ae38-0f9657672574" containerName="test-curl-7372-b3db15"
Sep 16 21:40:42.841031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.841011    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:42.841148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.841069    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:40:42.841148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.841132    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-e7bdb1"
Sep 16 21:40:42.844793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.844765    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7372-e7bdb1"]
Sep 16 21:40:42.938736 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:42.938700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7dcx\" (UniqueName: \"kubernetes.io/projected/9e732fd7-e21b-4412-8e73-77d5c9de9e16-kube-api-access-h7dcx\") pod \"test-curl-7372-e7bdb1\" (UID: \"9e732fd7-e21b-4412-8e73-77d5c9de9e16\") " pod="opendatahub/test-curl-7372-e7bdb1"
Sep 16 21:40:43.040120 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:43.040017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7dcx\" (UniqueName: \"kubernetes.io/projected/9e732fd7-e21b-4412-8e73-77d5c9de9e16-kube-api-access-h7dcx\") pod \"test-curl-7372-e7bdb1\" (UID: \"9e732fd7-e21b-4412-8e73-77d5c9de9e16\") " pod="opendatahub/test-curl-7372-e7bdb1"
Sep 16 21:40:43.048069 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:43.048034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7dcx\" (UniqueName: \"kubernetes.io/projected/9e732fd7-e21b-4412-8e73-77d5c9de9e16-kube-api-access-h7dcx\") pod \"test-curl-7372-e7bdb1\" (UID: \"9e732fd7-e21b-4412-8e73-77d5c9de9e16\") " pod="opendatahub/test-curl-7372-e7bdb1"
Sep 16 21:40:43.111980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:43.111945    2563 generic.go:356] "Generic (PLEG): container finished" podID="2a7ef5ed-383c-424e-b456-ff6c8794313e" containerID="0f63b8b03528cbf6f69acc36415549b16b2577ca7993bb35a8622a2f7e731b3e" exitCode=137
Sep 16 21:40:43.112460 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:43.111988    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-ec2834"
Sep 16 21:40:43.112460 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:43.112011    2563 scope.go:117] "RemoveContainer" containerID="0f63b8b03528cbf6f69acc36415549b16b2577ca7993bb35a8622a2f7e731b3e"
Sep 16 21:40:43.112460 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:43.112303    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7372-b3db15" container="test-curl-7372-b3db15"
Sep 16 21:40:43.120955 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:43.120932    2563 scope.go:117] "RemoveContainer" containerID="0f63b8b03528cbf6f69acc36415549b16b2577ca7993bb35a8622a2f7e731b3e"
Sep 16 21:40:43.121276 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:43.121256    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f63b8b03528cbf6f69acc36415549b16b2577ca7993bb35a8622a2f7e731b3e\": container with ID starting with 0f63b8b03528cbf6f69acc36415549b16b2577ca7993bb35a8622a2f7e731b3e not found: ID does not exist" containerID="0f63b8b03528cbf6f69acc36415549b16b2577ca7993bb35a8622a2f7e731b3e"
Sep 16 21:40:43.121339 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:43.121286    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f63b8b03528cbf6f69acc36415549b16b2577ca7993bb35a8622a2f7e731b3e"} err="failed to get container status \"0f63b8b03528cbf6f69acc36415549b16b2577ca7993bb35a8622a2f7e731b3e\": rpc error: code = NotFound desc = could not find container \"0f63b8b03528cbf6f69acc36415549b16b2577ca7993bb35a8622a2f7e731b3e\": container with ID starting with 0f63b8b03528cbf6f69acc36415549b16b2577ca7993bb35a8622a2f7e731b3e not found: ID does not exist"
Sep 16 21:40:43.152627 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:43.152584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-e7bdb1"
Sep 16 21:40:43.443590 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:43.443566    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7372-e7bdb1"]
Sep 16 21:40:43.445860 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:40:43.445829    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e732fd7_e21b_4412_8e73_77d5c9de9e16.slice/crio-54d35aebafd855429039a0cd3b77fa240e77eded4bf42f969ed7e52bb93915b4 WatchSource:0}: Error finding container 54d35aebafd855429039a0cd3b77fa240e77eded4bf42f969ed7e52bb93915b4: Status 404 returned error can't find the container with id 54d35aebafd855429039a0cd3b77fa240e77eded4bf42f969ed7e52bb93915b4
Sep 16 21:40:44.118560 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:44.118523    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7372-e7bdb1" event={"ID":"9e732fd7-e21b-4412-8e73-77d5c9de9e16","Type":"ContainerStarted","Data":"6ca289fbe5fc99158c01d2bc6eb22b4f56bd5fce1879146145a9a4b792ee1803"}
Sep 16 21:40:44.118988 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:44.118578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7372-e7bdb1" event={"ID":"9e732fd7-e21b-4412-8e73-77d5c9de9e16","Type":"ContainerStarted","Data":"54d35aebafd855429039a0cd3b77fa240e77eded4bf42f969ed7e52bb93915b4"}
Sep 16 21:40:44.119834 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:44.119679    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7372-b3db15" podUID="060d1891-ce8c-4961-ae38-0f9657672574" containerName="test-curl-7372-b3db15" containerID="cri-o://25cbd64d0fbb97286f7a849e7e31a8ea11473b09b5367bd83e27b4131fc8d87f" gracePeriod=2
Sep 16 21:40:44.131795 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:44.131748    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7372-e7bdb1" podStartSLOduration=2.13172846 podStartE2EDuration="2.13172846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:44.13139654 +0000 UTC m=+1104.031606544" watchObservedRunningTime="2026-09-16 21:40:44.13172846 +0000 UTC m=+1104.031938464"
Sep 16 21:40:44.133004 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:44.132976    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7372-b3db15\" is forbidden: User \"system:node:ip-10-0-136-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-118.ec2.internal' and this object" podUID="060d1891-ce8c-4961-ae38-0f9657672574" pod="opendatahub/test-curl-7372-b3db15"
Sep 16 21:40:44.322175 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:44.322149    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-442938"
Sep 16 21:40:44.324182 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:44.324151    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7372-b3db15\" is forbidden: User \"system:node:ip-10-0-136-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-118.ec2.internal' and this object" podUID="060d1891-ce8c-4961-ae38-0f9657672574" pod="opendatahub/test-curl-7372-b3db15"
Sep 16 21:40:44.453645 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:44.453615    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsxnb\" (UniqueName: \"kubernetes.io/projected/091a2527-3996-4e6e-8993-2c0a4e86d2e2-kube-api-access-rsxnb\") pod \"091a2527-3996-4e6e-8993-2c0a4e86d2e2\" (UID: \"091a2527-3996-4e6e-8993-2c0a4e86d2e2\") "
Sep 16 21:40:44.455731 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:44.455709    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091a2527-3996-4e6e-8993-2c0a4e86d2e2-kube-api-access-rsxnb" (OuterVolumeSpecName: "kube-api-access-rsxnb") pod "091a2527-3996-4e6e-8993-2c0a4e86d2e2" (UID: "091a2527-3996-4e6e-8993-2c0a4e86d2e2"). InnerVolumeSpecName "kube-api-access-rsxnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:40:44.555292 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:44.555257    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsxnb\" (UniqueName: \"kubernetes.io/projected/091a2527-3996-4e6e-8993-2c0a4e86d2e2-kube-api-access-rsxnb\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:44.608525 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:44.608490    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="091a2527-3996-4e6e-8993-2c0a4e86d2e2" path="/var/lib/kubelet/pods/091a2527-3996-4e6e-8993-2c0a4e86d2e2/volumes"
Sep 16 21:40:44.743446 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:44.743357    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7372-e7bdb1"]
Sep 16 21:40:44.747285 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:44.747255    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7372-e7bdb1"]
Sep 16 21:40:45.125841 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.125740    2563 generic.go:356] "Generic (PLEG): container finished" podID="091a2527-3996-4e6e-8993-2c0a4e86d2e2" containerID="162fd83d24dbf3f15c922d781f5d1695587701028a98916f30e1a5dd84e1cac3" exitCode=137
Sep 16 21:40:45.125841 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.125800    2563 scope.go:117] "RemoveContainer" containerID="162fd83d24dbf3f15c922d781f5d1695587701028a98916f30e1a5dd84e1cac3"
Sep 16 21:40:45.126402 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.125809    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-442938"
Sep 16 21:40:45.126402 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:45.126002    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7372-e7bdb1" container="test-curl-7372-e7bdb1"
Sep 16 21:40:45.134522 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.134501    2563 scope.go:117] "RemoveContainer" containerID="162fd83d24dbf3f15c922d781f5d1695587701028a98916f30e1a5dd84e1cac3"
Sep 16 21:40:45.134856 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:45.134836    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162fd83d24dbf3f15c922d781f5d1695587701028a98916f30e1a5dd84e1cac3\": container with ID starting with 162fd83d24dbf3f15c922d781f5d1695587701028a98916f30e1a5dd84e1cac3 not found: ID does not exist" containerID="162fd83d24dbf3f15c922d781f5d1695587701028a98916f30e1a5dd84e1cac3"
Sep 16 21:40:45.134909 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.134868    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162fd83d24dbf3f15c922d781f5d1695587701028a98916f30e1a5dd84e1cac3"} err="failed to get container status \"162fd83d24dbf3f15c922d781f5d1695587701028a98916f30e1a5dd84e1cac3\": rpc error: code = NotFound desc = could not find container \"162fd83d24dbf3f15c922d781f5d1695587701028a98916f30e1a5dd84e1cac3\": container with ID starting with 162fd83d24dbf3f15c922d781f5d1695587701028a98916f30e1a5dd84e1cac3 not found: ID does not exist"
Sep 16 21:40:45.621173 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.621139    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"]
Sep 16 21:40:45.621558 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.621543    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e732fd7-e21b-4412-8e73-77d5c9de9e16" containerName="test-curl-7372-e7bdb1"
Sep 16 21:40:45.621640 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.621559    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e732fd7-e21b-4412-8e73-77d5c9de9e16" containerName="test-curl-7372-e7bdb1"
Sep 16 21:40:45.621640 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.621610    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e732fd7-e21b-4412-8e73-77d5c9de9e16" containerName="test-curl-7372-e7bdb1"
Sep 16 21:40:45.626326 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.626300    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.628449 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.628421    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5ee44a89-openshift-default-dockercfg-cm4rb\""
Sep 16 21:40:45.636415 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.636385    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"]
Sep 16 21:40:45.680348 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.680291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istiod-ca-cert\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.690049 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.690010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-credential-socket\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.690327 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.690299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-workload-certs\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.690723 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.690697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vb8m\" (UniqueName: \"kubernetes.io/projected/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-kube-api-access-7vb8m\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.690808 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.690748    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-envoy\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.690808 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.690800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-workload-socket\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.691094 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.691052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-data\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.691161 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.691110    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-podinfo\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.691210 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.691155    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-token\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.792239 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.792089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-credential-socket\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.792239 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.792151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-workload-certs\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.792239 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.792169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vb8m\" (UniqueName: \"kubernetes.io/projected/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-kube-api-access-7vb8m\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.792239 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.792200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-envoy\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.792239 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.792238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-workload-socket\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.792696 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.792310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-data\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.792696 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.792332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-podinfo\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.792696 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.792356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-token\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.792696 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.792402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istiod-ca-cert\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.792696 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.792585    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-workload-certs\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.793217 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.793180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-workload-socket\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.793400 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.793374    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-data\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.793481 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.793397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-credential-socket\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.793757 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.793733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istiod-ca-cert\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.795322 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.795289    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-envoy\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.795322 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.795294    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-podinfo\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.801329 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.801277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-token\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.801459 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.801430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vb8m\" (UniqueName: \"kubernetes.io/projected/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-kube-api-access-7vb8m\") pod \"e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") " pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:45.938914 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:45.938880    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:46.131830 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:46.131797    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7372-e7bdb1" podUID="9e732fd7-e21b-4412-8e73-77d5c9de9e16" containerName="test-curl-7372-e7bdb1" containerID="cri-o://6ca289fbe5fc99158c01d2bc6eb22b4f56bd5fce1879146145a9a4b792ee1803" gracePeriod=2
Sep 16 21:40:46.133667 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:46.133632    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7372-e7bdb1\" is forbidden: User \"system:node:ip-10-0-136-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-118.ec2.internal' and this object" podUID="9e732fd7-e21b-4412-8e73-77d5c9de9e16" pod="opendatahub/test-curl-7372-e7bdb1"
Sep 16 21:40:46.242826 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:46.242789    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"]
Sep 16 21:40:46.244550 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:40:46.244513    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6434fe14_9fe2_40ee_98f8_cc96be3d2fb9.slice/crio-64510675eb7a0ff6abef95bba6050f2049255c0957fe0f3b8117b690d030819e WatchSource:0}: Error finding container 64510675eb7a0ff6abef95bba6050f2049255c0957fe0f3b8117b690d030819e: Status 404 returned error can't find the container with id 64510675eb7a0ff6abef95bba6050f2049255c0957fe0f3b8117b690d030819e
Sep 16 21:40:46.247042 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:46.247006    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:46.247132 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:46.247080    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:46.247132 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:46.247112    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:46.337440 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:46.337412    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-b3db15"
Sep 16 21:40:46.339472 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:46.339444    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7372-e7bdb1\" is forbidden: User \"system:node:ip-10-0-136-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-118.ec2.internal' and this object" podUID="9e732fd7-e21b-4412-8e73-77d5c9de9e16" pod="opendatahub/test-curl-7372-e7bdb1"
Sep 16 21:40:46.501119 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:46.501018    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd4v2\" (UniqueName: \"kubernetes.io/projected/060d1891-ce8c-4961-ae38-0f9657672574-kube-api-access-dd4v2\") pod \"060d1891-ce8c-4961-ae38-0f9657672574\" (UID: \"060d1891-ce8c-4961-ae38-0f9657672574\") "
Sep 16 21:40:46.503476 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:46.503437    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060d1891-ce8c-4961-ae38-0f9657672574-kube-api-access-dd4v2" (OuterVolumeSpecName: "kube-api-access-dd4v2") pod "060d1891-ce8c-4961-ae38-0f9657672574" (UID: "060d1891-ce8c-4961-ae38-0f9657672574"). InnerVolumeSpecName "kube-api-access-dd4v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:40:46.602928 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:46.602896    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd4v2\" (UniqueName: \"kubernetes.io/projected/060d1891-ce8c-4961-ae38-0f9657672574-kube-api-access-dd4v2\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:46.608564 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:46.608532    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="060d1891-ce8c-4961-ae38-0f9657672574" path="/var/lib/kubelet/pods/060d1891-ce8c-4961-ae38-0f9657672574/volumes"
Sep 16 21:40:47.139381 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:47.139288    2563 generic.go:356] "Generic (PLEG): container finished" podID="060d1891-ce8c-4961-ae38-0f9657672574" containerID="25cbd64d0fbb97286f7a849e7e31a8ea11473b09b5367bd83e27b4131fc8d87f" exitCode=137
Sep 16 21:40:47.139381 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:47.139327    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-b3db15"
Sep 16 21:40:47.139846 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:47.139383    2563 scope.go:117] "RemoveContainer" containerID="25cbd64d0fbb97286f7a849e7e31a8ea11473b09b5367bd83e27b4131fc8d87f"
Sep 16 21:40:47.141887 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:47.141408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s" event={"ID":"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9","Type":"ContainerStarted","Data":"2f841d592d134751eaaabdb858fd23b08ffc1ef5fb270a753090299d86102c9a"}
Sep 16 21:40:47.141887 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:47.141481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s" event={"ID":"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9","Type":"ContainerStarted","Data":"64510675eb7a0ff6abef95bba6050f2049255c0957fe0f3b8117b690d030819e"}
Sep 16 21:40:47.148686 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:47.148610    2563 scope.go:117] "RemoveContainer" containerID="25cbd64d0fbb97286f7a849e7e31a8ea11473b09b5367bd83e27b4131fc8d87f"
Sep 16 21:40:47.148998 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:47.148972    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25cbd64d0fbb97286f7a849e7e31a8ea11473b09b5367bd83e27b4131fc8d87f\": container with ID starting with 25cbd64d0fbb97286f7a849e7e31a8ea11473b09b5367bd83e27b4131fc8d87f not found: ID does not exist" containerID="25cbd64d0fbb97286f7a849e7e31a8ea11473b09b5367bd83e27b4131fc8d87f"
Sep 16 21:40:47.149050 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:47.149003    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25cbd64d0fbb97286f7a849e7e31a8ea11473b09b5367bd83e27b4131fc8d87f"} err="failed to get container status \"25cbd64d0fbb97286f7a849e7e31a8ea11473b09b5367bd83e27b4131fc8d87f\": rpc error: code = NotFound desc = could not find container \"25cbd64d0fbb97286f7a849e7e31a8ea11473b09b5367bd83e27b4131fc8d87f\": container with ID starting with 25cbd64d0fbb97286f7a849e7e31a8ea11473b09b5367bd83e27b4131fc8d87f not found: ID does not exist"
Sep 16 21:40:47.164152 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:47.164099    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s" podStartSLOduration=2.164078726 podStartE2EDuration="2.164078726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:47.16293569 +0000 UTC m=+1107.063145714" watchObservedRunningTime="2026-09-16 21:40:47.164078726 +0000 UTC m=+1107.064288798"
Sep 16 21:40:47.939009 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:47.938971    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:47.945012 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:47.944984    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:48.148498 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:48.148470    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:48.149395 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:48.149376    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:40:48.345314 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:48.345289    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-e7bdb1"
Sep 16 21:40:48.422487 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:48.422453    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7dcx\" (UniqueName: \"kubernetes.io/projected/9e732fd7-e21b-4412-8e73-77d5c9de9e16-kube-api-access-h7dcx\") pod \"9e732fd7-e21b-4412-8e73-77d5c9de9e16\" (UID: \"9e732fd7-e21b-4412-8e73-77d5c9de9e16\") "
Sep 16 21:40:48.424636 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:48.424601    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e732fd7-e21b-4412-8e73-77d5c9de9e16-kube-api-access-h7dcx" (OuterVolumeSpecName: "kube-api-access-h7dcx") pod "9e732fd7-e21b-4412-8e73-77d5c9de9e16" (UID: "9e732fd7-e21b-4412-8e73-77d5c9de9e16"). InnerVolumeSpecName "kube-api-access-h7dcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:40:48.524178 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:48.524076    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7dcx\" (UniqueName: \"kubernetes.io/projected/9e732fd7-e21b-4412-8e73-77d5c9de9e16-kube-api-access-h7dcx\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:40:48.609971 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:48.609934    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e732fd7-e21b-4412-8e73-77d5c9de9e16" path="/var/lib/kubelet/pods/9e732fd7-e21b-4412-8e73-77d5c9de9e16/volumes"
Sep 16 21:40:49.156632 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:49.156600    2563 generic.go:356] "Generic (PLEG): container finished" podID="9e732fd7-e21b-4412-8e73-77d5c9de9e16" containerID="6ca289fbe5fc99158c01d2bc6eb22b4f56bd5fce1879146145a9a4b792ee1803" exitCode=137
Sep 16 21:40:49.157062 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:49.156670    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7372-e7bdb1"
Sep 16 21:40:49.157062 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:49.156702    2563 scope.go:117] "RemoveContainer" containerID="6ca289fbe5fc99158c01d2bc6eb22b4f56bd5fce1879146145a9a4b792ee1803"
Sep 16 21:40:49.168076 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:49.167781    2563 scope.go:117] "RemoveContainer" containerID="6ca289fbe5fc99158c01d2bc6eb22b4f56bd5fce1879146145a9a4b792ee1803"
Sep 16 21:40:49.168220 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:40:49.168197    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca289fbe5fc99158c01d2bc6eb22b4f56bd5fce1879146145a9a4b792ee1803\": container with ID starting with 6ca289fbe5fc99158c01d2bc6eb22b4f56bd5fce1879146145a9a4b792ee1803 not found: ID does not exist" containerID="6ca289fbe5fc99158c01d2bc6eb22b4f56bd5fce1879146145a9a4b792ee1803"
Sep 16 21:40:49.168279 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:49.168229    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca289fbe5fc99158c01d2bc6eb22b4f56bd5fce1879146145a9a4b792ee1803"} err="failed to get container status \"6ca289fbe5fc99158c01d2bc6eb22b4f56bd5fce1879146145a9a4b792ee1803\": rpc error: code = NotFound desc = could not find container \"6ca289fbe5fc99158c01d2bc6eb22b4f56bd5fce1879146145a9a4b792ee1803\": container with ID starting with 6ca289fbe5fc99158c01d2bc6eb22b4f56bd5fce1879146145a9a4b792ee1803 not found: ID does not exist"
Sep 16 21:40:52.177178 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:52.177142    2563 generic.go:356] "Generic (PLEG): container finished" podID="8fb9ef5a-b300-49de-8994-39e07ce01508" containerID="5f5d3140496b8c8119b4316029763adc3e2343b391104dac3f83b68fd73167c1" exitCode=0
Sep 16 21:40:52.177750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:52.177236    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t" event={"ID":"8fb9ef5a-b300-49de-8994-39e07ce01508","Type":"ContainerDied","Data":"5f5d3140496b8c8119b4316029763adc3e2343b391104dac3f83b68fd73167c1"}
Sep 16 21:40:52.177750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:52.177711    2563 scope.go:117] "RemoveContainer" containerID="5f5d3140496b8c8119b4316029763adc3e2343b391104dac3f83b68fd73167c1"
Sep 16 21:40:52.178866 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:52.178831    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:52.178958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:52.178903    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:52.178958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:52.178935    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:53.185745 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:53.185707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t" event={"ID":"8fb9ef5a-b300-49de-8994-39e07ce01508","Type":"ContainerStarted","Data":"b7f3df3b4b2a19b4974a9536d4042196420017f1a51ac010902ed43ca8c7b23c"}
Sep 16 21:40:53.303453 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:53.303409    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:53.304601 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:53.304571    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 16 21:40:53.304733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:53.304623    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t" podUID="8fb9ef5a-b300-49de-8994-39e07ce01508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 16 21:40:53.980675 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:53.980612    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:54.304545 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:54.304449    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 16 21:40:54.305176 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:54.305146    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t" podUID="8fb9ef5a-b300-49de-8994-39e07ce01508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 16 21:40:55.304206 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.304167    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 16 21:40:55.304385 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.304235    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t" podUID="8fb9ef5a-b300-49de-8994-39e07ce01508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 16 21:40:55.654936 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.654840    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"]
Sep 16 21:40:55.658896 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.658868    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.661236 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.661206    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-1b68db-openshift-default-dockercfg-qtpss\""
Sep 16 21:40:55.684358 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.672638    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"]
Sep 16 21:40:55.710518 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.710484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7wq8\" (UniqueName: \"kubernetes.io/projected/49ea78ff-4414-4ffa-9791-8248d6f43535-kube-api-access-p7wq8\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.710785 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.710551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-data\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.710785 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.710586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-workload-certs\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.710785 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.710637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-workload-socket\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.710976 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.710794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-token\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.710976 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.710860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-podinfo\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.710976 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.710889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-envoy\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.711122 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.710990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ea78ff-4414-4ffa-9791-8248d6f43535-istiod-ca-cert\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.711122 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.711016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-credential-socket\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.813681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.811897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ea78ff-4414-4ffa-9791-8248d6f43535-istiod-ca-cert\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.813681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.811954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-credential-socket\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.813681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.812019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7wq8\" (UniqueName: \"kubernetes.io/projected/49ea78ff-4414-4ffa-9791-8248d6f43535-kube-api-access-p7wq8\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.813681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.812075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-data\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.813681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.812108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-workload-certs\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.813681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.812180    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-workload-socket\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.813681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.812219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-token\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.813681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.812283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-podinfo\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.813681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.812313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-envoy\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.813681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.812578    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-credential-socket\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.813681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.812781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ea78ff-4414-4ffa-9791-8248d6f43535-istiod-ca-cert\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.813681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.812955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-workload-certs\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.813681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.813336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-workload-socket\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.813681 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.813498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-data\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.817968 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.817940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-envoy\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.823254 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.823195    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-podinfo\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.824356 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.824310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-token\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.827368 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.827320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7wq8\" (UniqueName: \"kubernetes.io/projected/49ea78ff-4414-4ffa-9791-8248d6f43535-kube-api-access-p7wq8\") pod \"e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") " pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:55.986082 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:55.986038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:56.275766 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:56.273612    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"]
Sep 16 21:40:56.281045 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:40:56.280991    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ea78ff_4414_4ffa_9791_8248d6f43535.slice/crio-663650b6b2ece1d89728aa08be61194297c74f32bd3d2c4ce9012e3ab7ca01cd WatchSource:0}: Error finding container 663650b6b2ece1d89728aa08be61194297c74f32bd3d2c4ce9012e3ab7ca01cd: Status 404 returned error can't find the container with id 663650b6b2ece1d89728aa08be61194297c74f32bd3d2c4ce9012e3ab7ca01cd
Sep 16 21:40:56.284499 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:56.283448    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:56.284499 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:56.283544    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:56.284499 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:56.283584    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:40:56.304350 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:56.304319    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 16 21:40:56.304467 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:56.304391    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t" podUID="8fb9ef5a-b300-49de-8994-39e07ce01508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 16 21:40:57.220688 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:57.220630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4" event={"ID":"49ea78ff-4414-4ffa-9791-8248d6f43535","Type":"ContainerStarted","Data":"5ea55bab54e14c067af4c5d52963ca415eba38d93745ecc6e923f81082de5650"}
Sep 16 21:40:57.221170 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:57.220697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4" event={"ID":"49ea78ff-4414-4ffa-9791-8248d6f43535","Type":"ContainerStarted","Data":"663650b6b2ece1d89728aa08be61194297c74f32bd3d2c4ce9012e3ab7ca01cd"}
Sep 16 21:40:57.241446 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:57.241383    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4" podStartSLOduration=2.24136638 podStartE2EDuration="2.24136638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:57.239690164 +0000 UTC m=+1117.139900170" watchObservedRunningTime="2026-09-16 21:40:57.24136638 +0000 UTC m=+1117.141576384"
Sep 16 21:40:57.304317 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:57.304279    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 16 21:40:57.304488 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:57.304340    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t" podUID="8fb9ef5a-b300-49de-8994-39e07ce01508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 16 21:40:57.989075 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:57.989033    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:57.993794 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:57.993769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:58.227040 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:58.227011    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:58.228144 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:58.228124    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:40:58.303796 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:58.303696    2563 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 16 21:40:58.303796 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:58.303774    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t" podUID="8fb9ef5a-b300-49de-8994-39e07ce01508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 16 21:40:59.431817 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:59.431768    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:40:59.433283 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:40:59.433260    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:41:18.150070 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:18.150022    2563 patch_prober.go:28] interesting pod/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 16 21:41:18.150631 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:18.150107    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s" podUID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 16 21:41:18.366264 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:18.366221    2563 generic.go:356] "Generic (PLEG): container finished" podID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" containerID="2f841d592d134751eaaabdb858fd23b08ffc1ef5fb270a753090299d86102c9a" exitCode=0
Sep 16 21:41:18.368331 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:18.366274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s" event={"ID":"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9","Type":"ContainerDied","Data":"2f841d592d134751eaaabdb858fd23b08ffc1ef5fb270a753090299d86102c9a"}
Sep 16 21:41:18.368331 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:18.366832    2563 scope.go:117] "RemoveContainer" containerID="2f841d592d134751eaaabdb858fd23b08ffc1ef5fb270a753090299d86102c9a"
Sep 16 21:41:18.368331 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:18.368056    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:41:18.368331 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:18.368130    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:41:18.368331 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:18.368169    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:41:19.378621 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:19.378564    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s" event={"ID":"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9","Type":"ContainerStarted","Data":"273078871c4f3d0792c95755992f49114f781ef8c55ff9577d2830956fda32dc"}
Sep 16 21:41:19.384085 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:19.384050    2563 generic.go:356] "Generic (PLEG): container finished" podID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerID="5ea55bab54e14c067af4c5d52963ca415eba38d93745ecc6e923f81082de5650" exitCode=0
Sep 16 21:41:19.384735 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:19.384323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4" event={"ID":"49ea78ff-4414-4ffa-9791-8248d6f43535","Type":"ContainerDied","Data":"5ea55bab54e14c067af4c5d52963ca415eba38d93745ecc6e923f81082de5650"}
Sep 16 21:41:19.385387 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:19.385372    2563 scope.go:117] "RemoveContainer" containerID="5ea55bab54e14c067af4c5d52963ca415eba38d93745ecc6e923f81082de5650"
Sep 16 21:41:19.393839 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:19.393795    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:41:19.394000 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:19.393890    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:41:19.394000 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:19.393938    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:41:19.939104 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:19.939063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:41:19.939389 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:19.939365    2563 patch_prober.go:28] interesting pod/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 16 21:41:19.939445 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:19.939426    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s" podUID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 16 21:41:20.397841 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:20.397741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4" event={"ID":"49ea78ff-4414-4ffa-9791-8248d6f43535","Type":"ContainerStarted","Data":"385f1e757e30eb02daed326d2bdbc9ba6e4a66e2b9b312555513946876f938f8"}
Sep 16 21:41:20.706684 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:20.701257    2563 scope.go:117] "RemoveContainer" containerID="dd35502c37084fd6c2d6f4ac266e74b3b5f26dbfc87b3f4f25dfb693bbf24737"
Sep 16 21:41:20.939713 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:20.939672    2563 patch_prober.go:28] interesting pod/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 16 21:41:20.939908 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:20.939732    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s" podUID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 16 21:41:20.987240 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:20.987132    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:41:20.987424 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:20.987395    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 16 21:41:20.987501 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:20.987462    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4" podUID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 16 21:41:21.939885 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:21.939843    2563 patch_prober.go:28] interesting pod/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 16 21:41:21.940401 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:21.939925    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s" podUID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 16 21:41:21.987963 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:21.987911    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 16 21:41:21.988152 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:21.987987    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4" podUID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 16 21:41:22.939944 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:22.939912    2563 patch_prober.go:28] interesting pod/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 16 21:41:22.940398 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:22.940373    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s" podUID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 16 21:41:22.986938 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:22.986897    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 16 21:41:22.987231 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:22.987198    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4" podUID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 16 21:41:23.939678 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:23.939615    2563 patch_prober.go:28] interesting pod/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 16 21:41:23.940101 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:23.940062    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s" podUID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 16 21:41:23.986858 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:23.986816    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 16 21:41:23.987064 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:23.986881    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4" podUID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 16 21:41:24.943963 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:24.943929    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:41:24.944326 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:24.944198    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:41:24.945064 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:24.945042    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:41:24.987290 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:24.987240    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 16 21:41:24.987499 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:24.987329    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4" podUID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 16 21:41:25.987099 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:25.987049    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 16 21:41:25.987496 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:25.987132    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4" podUID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 16 21:41:26.990973 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:26.990943    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:41:26.991386 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:26.991343    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:41:26.991986 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:26.991965    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:41:45.881188 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:45.881109    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"]
Sep 16 21:41:45.886278 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:45.886253    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:45.888845 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:45.888816    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-35f5fd\"/\"distinct-2-w0-35f5fd-kserve-self-signed-certs\""
Sep 16 21:41:45.895056 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:45.895031    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"]
Sep 16 21:41:45.899814 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:45.899778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-tmp-dir\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:45.899929 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:45.899822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g2l5\" (UniqueName: \"kubernetes.io/projected/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-kube-api-access-7g2l5\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:45.899929 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:45.899884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-home\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:45.900002 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:45.899929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-dshm\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:45.900002 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:45.899979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-model-cache\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:45.900102 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:45.900081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-tls-certs\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:46.000738 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.000697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-tls-certs\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:46.000926 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.000785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-tmp-dir\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:46.000926 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.000803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g2l5\" (UniqueName: \"kubernetes.io/projected/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-kube-api-access-7g2l5\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:46.000926 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.000825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-home\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:46.000926 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.000909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-dshm\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:46.001119 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.000985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-model-cache\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:46.001293 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.001263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-tmp-dir\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:46.001364 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.001297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-home\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:46.001364 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.001333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-model-cache\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:46.003219 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.003201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-dshm\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:46.003349 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.003334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-tls-certs\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:46.008961 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.008936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g2l5\" (UniqueName: \"kubernetes.io/projected/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-kube-api-access-7g2l5\") pod \"distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:46.197499 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.197460    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:46.471126 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.471100    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"]
Sep 16 21:41:46.473404 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:41:46.473372    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdaf67ad_cce8_4dfe_9f82_48df7fcaed88.slice/crio-61ad5c7192324eb507f7e2284dd0e9a220249eeb51d26126a6f7b5b7a85176fb WatchSource:0}: Error finding container 61ad5c7192324eb507f7e2284dd0e9a220249eeb51d26126a6f7b5b7a85176fb: Status 404 returned error can't find the container with id 61ad5c7192324eb507f7e2284dd0e9a220249eeb51d26126a6f7b5b7a85176fb
Sep 16 21:41:46.549795 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.549761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28" event={"ID":"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88","Type":"ContainerStarted","Data":"25d20f95e1a8f6d6c11a3ad833b520bbb863ad08fe1d3c9d48b7647edcbe634d"}
Sep 16 21:41:46.549795 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.549802    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28" event={"ID":"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88","Type":"ContainerStarted","Data":"61ad5c7192324eb507f7e2284dd0e9a220249eeb51d26126a6f7b5b7a85176fb"}
Sep 16 21:41:46.565784 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:46.565727    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28" podStartSLOduration=1.565709308 podStartE2EDuration="1.565709308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:41:46.565146988 +0000 UTC m=+1166.465356992" watchObservedRunningTime="2026-09-16 21:41:46.565709308 +0000 UTC m=+1166.465919312"
Sep 16 21:41:56.198698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:56.198629    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:56.199218 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:56.198725    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:56.211045 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:56.211014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:56.617857 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:56.617776    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:41:57.226623 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.226591    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"]
Sep 16 21:41:57.229902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.229879    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.233021 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.232988    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5ee44a89\"/\"kube-root-ca.crt\""
Sep 16 21:41:57.233128 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.233062    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5ee44a89\"/\"e2e-lifecycle-model-5ee44a89-kserve-self-signed-certs\""
Sep 16 21:41:57.233329 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.233314    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5ee44a89\"/\"default-dockercfg-54dbv\""
Sep 16 21:41:57.233385 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.233332    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5ee44a89\"/\"openshift-service-ca.crt\""
Sep 16 21:41:57.240153 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.240131    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"]
Sep 16 21:41:57.317543 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.317511    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-tmp-dir\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.317543 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.317558    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-home\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.317811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.317627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9gfv\" (UniqueName: \"kubernetes.io/projected/3716ae16-a9c5-410d-821f-fde454c463db-kube-api-access-c9gfv\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.317811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.317754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3716ae16-a9c5-410d-821f-fde454c463db-tls-certs\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.317811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.317792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-dshm\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.317918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.317814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-model-cache\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.418901 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.418856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9gfv\" (UniqueName: \"kubernetes.io/projected/3716ae16-a9c5-410d-821f-fde454c463db-kube-api-access-c9gfv\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.419108 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.418953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3716ae16-a9c5-410d-821f-fde454c463db-tls-certs\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.419108 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.418989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-dshm\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.419108 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.419010    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-model-cache\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.419108 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.419060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-tmp-dir\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.419108 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.419091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-home\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.419488 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.419462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-home\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.419626 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.419514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-tmp-dir\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.419626 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.419564    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-model-cache\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.421438 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.421404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-dshm\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.421554 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.421502    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3716ae16-a9c5-410d-821f-fde454c463db-tls-certs\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.426406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.426383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9gfv\" (UniqueName: \"kubernetes.io/projected/3716ae16-a9c5-410d-821f-fde454c463db-kube-api-access-c9gfv\") pod \"e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") " pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.540774 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.540648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:41:57.813411 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:57.813327    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"]
Sep 16 21:41:57.816525 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:41:57.816496    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3716ae16_a9c5_410d_821f_fde454c463db.slice/crio-ff9b1be1c0a9e52526ec9d0a37b4cc10393f6afb6dfd60ad621a67932f2c0e2e WatchSource:0}: Error finding container ff9b1be1c0a9e52526ec9d0a37b4cc10393f6afb6dfd60ad621a67932f2c0e2e: Status 404 returned error can't find the container with id ff9b1be1c0a9e52526ec9d0a37b4cc10393f6afb6dfd60ad621a67932f2c0e2e
Sep 16 21:41:58.619799 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:58.619760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8" event={"ID":"3716ae16-a9c5-410d-821f-fde454c463db","Type":"ContainerStarted","Data":"6ce406ebc51c4058e469ba2dd360de29fb953c9b7af4bf3856f0b394b92bfa8b"}
Sep 16 21:41:58.620178 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:58.619808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8" event={"ID":"3716ae16-a9c5-410d-821f-fde454c463db","Type":"ContainerStarted","Data":"ff9b1be1c0a9e52526ec9d0a37b4cc10393f6afb6dfd60ad621a67932f2c0e2e"}
Sep 16 21:41:58.638976 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:41:58.638926    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8" podStartSLOduration=1.6389095089999999 podStartE2EDuration="1.638909509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:41:58.63807638 +0000 UTC m=+1178.538286384" watchObservedRunningTime="2026-09-16 21:41:58.638909509 +0000 UTC m=+1178.539119512"
Sep 16 21:42:05.180482 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.180445    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"]
Sep 16 21:42:05.184262 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.184240    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.186571 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.186548    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7df81\"/\"openshift-service-ca.crt\""
Sep 16 21:42:05.187329 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.187307    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7df81\"/\"auth-test-model-a7df81-kserve-self-signed-certs\""
Sep 16 21:42:05.187329 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.187325    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7df81\"/\"kube-root-ca.crt\""
Sep 16 21:42:05.187721 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.187708    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7df81\"/\"default-dockercfg-vvzqj\""
Sep 16 21:42:05.196159 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.196126    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"]
Sep 16 21:42:05.309110 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.309074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-home\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.309110 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.309119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-model-cache\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.309391 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.309140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-tmp-dir\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.309391 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.309237    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkvdz\" (UniqueName: \"kubernetes.io/projected/2816143a-03ad-414b-b842-fb6b4b76984c-kube-api-access-jkvdz\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.309391 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.309327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-dshm\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.309565 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.309443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2816143a-03ad-414b-b842-fb6b4b76984c-tls-certs\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.410744 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.410698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2816143a-03ad-414b-b842-fb6b4b76984c-tls-certs\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.410953 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.410816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-home\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.410953 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.410851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-model-cache\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.410953 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.410872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-tmp-dir\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.410953 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.410931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkvdz\" (UniqueName: \"kubernetes.io/projected/2816143a-03ad-414b-b842-fb6b4b76984c-kube-api-access-jkvdz\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.411164 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.410975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-dshm\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.411268 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.411242    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-home\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.411339 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.411313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-tmp-dir\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.411391 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.411341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-model-cache\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.413153 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.413122    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-dshm\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.413337 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.413321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2816143a-03ad-414b-b842-fb6b4b76984c-tls-certs\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.418909 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.418879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkvdz\" (UniqueName: \"kubernetes.io/projected/2816143a-03ad-414b-b842-fb6b4b76984c-kube-api-access-jkvdz\") pod \"auth-test-model-a7df81-kserve-8467d686d7-czj96\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.496924 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.496829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:05.772730 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:05.772704    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"]
Sep 16 21:42:05.774947 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:42:05.774905    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2816143a_03ad_414b_b842_fb6b4b76984c.slice/crio-5c3996f666ee2b1a4a91a198d57468c001bdff846e6eb2002c6cf5037bfac868 WatchSource:0}: Error finding container 5c3996f666ee2b1a4a91a198d57468c001bdff846e6eb2002c6cf5037bfac868: Status 404 returned error can't find the container with id 5c3996f666ee2b1a4a91a198d57468c001bdff846e6eb2002c6cf5037bfac868
Sep 16 21:42:06.668275 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:06.668238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96" event={"ID":"2816143a-03ad-414b-b842-fb6b4b76984c","Type":"ContainerStarted","Data":"ab0571d31596589f3f38ee7a249014a5ce2da3ee80697a1385513a293ecbaf85"}
Sep 16 21:42:06.668914 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:06.668287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96" event={"ID":"2816143a-03ad-414b-b842-fb6b4b76984c","Type":"ContainerStarted","Data":"5c3996f666ee2b1a4a91a198d57468c001bdff846e6eb2002c6cf5037bfac868"}
Sep 16 21:42:06.684593 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:06.684538    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96" podStartSLOduration=1.684522294 podStartE2EDuration="1.684522294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:42:06.682835114 +0000 UTC m=+1186.583045126" watchObservedRunningTime="2026-09-16 21:42:06.684522294 +0000 UTC m=+1186.584732315"
Sep 16 21:42:07.541165 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:07.541128    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:42:07.541444 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:07.541402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:42:07.554128 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:07.554103    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:42:07.684998 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:07.684968    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:42:15.497766 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:15.497676    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:15.497766 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:15.497731    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:15.510381 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:15.510345    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:15.731030 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:15.730998    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:42:20.761526 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:20.761495    2563 scope.go:117] "RemoveContainer" containerID="896c960b317a6f5e1623c02b49bbce8363e2d797b68d88df775cef32cb896f65"
Sep 16 21:42:20.770438 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:20.770416    2563 scope.go:117] "RemoveContainer" containerID="f42e3c6663d923270d8032e9000c81bc386f5d6c4c95aa3e9610b0de513770bc"
Sep 16 21:42:20.778518 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:42:20.778498    2563 scope.go:117] "RemoveContainer" containerID="813d57a1b98faae2565c58dc89bbbacffb547d05d48b01729ab9d456a27d5884"
Sep 16 21:43:30.078328 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:30.078291    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"]
Sep 16 21:43:45.098155 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.098109    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8" podUID="3716ae16-a9c5-410d-821f-fde454c463db" containerName="main" containerID="cri-o://6ce406ebc51c4058e469ba2dd360de29fb953c9b7af4bf3856f0b394b92bfa8b" gracePeriod=45
Sep 16 21:43:45.229725 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.229694    2563 generic.go:356] "Generic (PLEG): container finished" podID="3716ae16-a9c5-410d-821f-fde454c463db" containerID="6ce406ebc51c4058e469ba2dd360de29fb953c9b7af4bf3856f0b394b92bfa8b" exitCode=0
Sep 16 21:43:45.229890 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.229779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8" event={"ID":"3716ae16-a9c5-410d-821f-fde454c463db","Type":"ContainerDied","Data":"6ce406ebc51c4058e469ba2dd360de29fb953c9b7af4bf3856f0b394b92bfa8b"}
Sep 16 21:43:45.357529 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.357456    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:43:45.458472 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.458439    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-dshm\") pod \"3716ae16-a9c5-410d-821f-fde454c463db\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") "
Sep 16 21:43:45.458642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.458493    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-model-cache\") pod \"3716ae16-a9c5-410d-821f-fde454c463db\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") "
Sep 16 21:43:45.458642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.458526    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-home\") pod \"3716ae16-a9c5-410d-821f-fde454c463db\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") "
Sep 16 21:43:45.458642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.458571    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3716ae16-a9c5-410d-821f-fde454c463db-tls-certs\") pod \"3716ae16-a9c5-410d-821f-fde454c463db\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") "
Sep 16 21:43:45.458642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.458599    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-tmp-dir\") pod \"3716ae16-a9c5-410d-821f-fde454c463db\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") "
Sep 16 21:43:45.458642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.458630    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9gfv\" (UniqueName: \"kubernetes.io/projected/3716ae16-a9c5-410d-821f-fde454c463db-kube-api-access-c9gfv\") pod \"3716ae16-a9c5-410d-821f-fde454c463db\" (UID: \"3716ae16-a9c5-410d-821f-fde454c463db\") "
Sep 16 21:43:45.458907 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.458860    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-home" (OuterVolumeSpecName: "home") pod "3716ae16-a9c5-410d-821f-fde454c463db" (UID: "3716ae16-a9c5-410d-821f-fde454c463db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:43:45.458980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.458946    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-model-cache" (OuterVolumeSpecName: "model-cache") pod "3716ae16-a9c5-410d-821f-fde454c463db" (UID: "3716ae16-a9c5-410d-821f-fde454c463db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:43:45.459036 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.458958    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3716ae16-a9c5-410d-821f-fde454c463db" (UID: "3716ae16-a9c5-410d-821f-fde454c463db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:43:45.459099 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.459085    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-model-cache\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:43:45.459154 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.459101    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-home\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:43:45.459154 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.459111    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-tmp-dir\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:43:45.460904 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.460878    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-dshm" (OuterVolumeSpecName: "dshm") pod "3716ae16-a9c5-410d-821f-fde454c463db" (UID: "3716ae16-a9c5-410d-821f-fde454c463db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:43:45.461031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.460925    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3716ae16-a9c5-410d-821f-fde454c463db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3716ae16-a9c5-410d-821f-fde454c463db" (UID: "3716ae16-a9c5-410d-821f-fde454c463db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:43:45.461031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.460978    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3716ae16-a9c5-410d-821f-fde454c463db-kube-api-access-c9gfv" (OuterVolumeSpecName: "kube-api-access-c9gfv") pod "3716ae16-a9c5-410d-821f-fde454c463db" (UID: "3716ae16-a9c5-410d-821f-fde454c463db"). InnerVolumeSpecName "kube-api-access-c9gfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:43:45.559760 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.559722    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3716ae16-a9c5-410d-821f-fde454c463db-dshm\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:43:45.559760 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.559753    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3716ae16-a9c5-410d-821f-fde454c463db-tls-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:43:45.559760 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:45.559764    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9gfv\" (UniqueName: \"kubernetes.io/projected/3716ae16-a9c5-410d-821f-fde454c463db-kube-api-access-c9gfv\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:43:46.236346 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:46.236308    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8" event={"ID":"3716ae16-a9c5-410d-821f-fde454c463db","Type":"ContainerDied","Data":"ff9b1be1c0a9e52526ec9d0a37b4cc10393f6afb6dfd60ad621a67932f2c0e2e"}
Sep 16 21:43:46.236794 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:46.236355    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"
Sep 16 21:43:46.236794 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:46.236363    2563 scope.go:117] "RemoveContainer" containerID="6ce406ebc51c4058e469ba2dd360de29fb953c9b7af4bf3856f0b394b92bfa8b"
Sep 16 21:43:46.257427 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:46.257399    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"]
Sep 16 21:43:46.261380 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:46.261355    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5ee44a89/e2e-lifecycle-model-5ee44a89-kserve-684d4b577f-sgts8"]
Sep 16 21:43:46.610026 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:43:46.609944    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3716ae16-a9c5-410d-821f-fde454c463db" path="/var/lib/kubelet/pods/3716ae16-a9c5-410d-821f-fde454c463db/volumes"
Sep 16 21:44:12.963410 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:12.963369    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"]
Sep 16 21:44:12.963828 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:12.963641    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s" podUID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" containerName="istio-proxy" containerID="cri-o://273078871c4f3d0792c95755992f49114f781ef8c55ff9577d2830956fda32dc" gracePeriod=30
Sep 16 21:44:18.219785 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.219758    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:44:18.317741 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.317627    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-token\") pod \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") "
Sep 16 21:44:18.317741 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.317687    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vb8m\" (UniqueName: \"kubernetes.io/projected/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-kube-api-access-7vb8m\") pod \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") "
Sep 16 21:44:18.317741 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.317724    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-credential-socket\") pod \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") "
Sep 16 21:44:18.318025 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.317754    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istiod-ca-cert\") pod \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") "
Sep 16 21:44:18.318025 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.317784    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-data\") pod \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") "
Sep 16 21:44:18.318025 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.317806    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-workload-certs\") pod \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") "
Sep 16 21:44:18.318025 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.317853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-workload-socket\") pod \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") "
Sep 16 21:44:18.318025 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.317882    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-podinfo\") pod \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") "
Sep 16 21:44:18.318025 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.317954    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-envoy\") pod \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\" (UID: \"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9\") "
Sep 16 21:44:18.318343 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.318089    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" (UID: "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:44:18.318343 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.318099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-data" (OuterVolumeSpecName: "istio-data") pod "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" (UID: "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:44:18.318343 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.318195    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" (UID: "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:44:18.318493 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.318338    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" (UID: "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:44:18.318543 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.318510    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:44:18.318543 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.318531    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:44:18.318633 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.318545    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:44:18.318633 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.318560    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:44:18.318732 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.318635    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" (UID: "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:44:18.319971 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.319944    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-token" (OuterVolumeSpecName: "istio-token") pod "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" (UID: "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:44:18.320079 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.320002    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-kube-api-access-7vb8m" (OuterVolumeSpecName: "kube-api-access-7vb8m") pod "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" (UID: "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9"). InnerVolumeSpecName "kube-api-access-7vb8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:44:18.320449 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.320431    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" (UID: "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:44:18.320515 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.320442    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" (UID: "6434fe14-9fe2-40ee-98f8-cc96be3d2fb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:44:18.408841 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.408806    2563 generic.go:356] "Generic (PLEG): container finished" podID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" containerID="273078871c4f3d0792c95755992f49114f781ef8c55ff9577d2830956fda32dc" exitCode=0
Sep 16 21:44:18.408841 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.408845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s" event={"ID":"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9","Type":"ContainerDied","Data":"273078871c4f3d0792c95755992f49114f781ef8c55ff9577d2830956fda32dc"}
Sep 16 21:44:18.409066 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.408854    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"
Sep 16 21:44:18.409066 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.408889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s" event={"ID":"6434fe14-9fe2-40ee-98f8-cc96be3d2fb9","Type":"ContainerDied","Data":"64510675eb7a0ff6abef95bba6050f2049255c0957fe0f3b8117b690d030819e"}
Sep 16 21:44:18.409066 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.408912    2563 scope.go:117] "RemoveContainer" containerID="273078871c4f3d0792c95755992f49114f781ef8c55ff9577d2830956fda32dc"
Sep 16 21:44:18.418471 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.418454    2563 scope.go:117] "RemoveContainer" containerID="2f841d592d134751eaaabdb858fd23b08ffc1ef5fb270a753090299d86102c9a"
Sep 16 21:44:18.419227 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.419209    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:44:18.419314 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.419230    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:44:18.419314 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.419240    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vb8m\" (UniqueName: \"kubernetes.io/projected/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-kube-api-access-7vb8m\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:44:18.419314 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.419252    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:44:18.419314 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.419260    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:44:18.426559 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.426543    2563 scope.go:117] "RemoveContainer" containerID="273078871c4f3d0792c95755992f49114f781ef8c55ff9577d2830956fda32dc"
Sep 16 21:44:18.426869 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:44:18.426847    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273078871c4f3d0792c95755992f49114f781ef8c55ff9577d2830956fda32dc\": container with ID starting with 273078871c4f3d0792c95755992f49114f781ef8c55ff9577d2830956fda32dc not found: ID does not exist" containerID="273078871c4f3d0792c95755992f49114f781ef8c55ff9577d2830956fda32dc"
Sep 16 21:44:18.426957 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.426875    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273078871c4f3d0792c95755992f49114f781ef8c55ff9577d2830956fda32dc"} err="failed to get container status \"273078871c4f3d0792c95755992f49114f781ef8c55ff9577d2830956fda32dc\": rpc error: code = NotFound desc = could not find container \"273078871c4f3d0792c95755992f49114f781ef8c55ff9577d2830956fda32dc\": container with ID starting with 273078871c4f3d0792c95755992f49114f781ef8c55ff9577d2830956fda32dc not found: ID does not exist"
Sep 16 21:44:18.426957 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.426892    2563 scope.go:117] "RemoveContainer" containerID="2f841d592d134751eaaabdb858fd23b08ffc1ef5fb270a753090299d86102c9a"
Sep 16 21:44:18.427139 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:44:18.427121    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f841d592d134751eaaabdb858fd23b08ffc1ef5fb270a753090299d86102c9a\": container with ID starting with 2f841d592d134751eaaabdb858fd23b08ffc1ef5fb270a753090299d86102c9a not found: ID does not exist" containerID="2f841d592d134751eaaabdb858fd23b08ffc1ef5fb270a753090299d86102c9a"
Sep 16 21:44:18.427180 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.427144    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f841d592d134751eaaabdb858fd23b08ffc1ef5fb270a753090299d86102c9a"} err="failed to get container status \"2f841d592d134751eaaabdb858fd23b08ffc1ef5fb270a753090299d86102c9a\": rpc error: code = NotFound desc = could not find container \"2f841d592d134751eaaabdb858fd23b08ffc1ef5fb270a753090299d86102c9a\": container with ID starting with 2f841d592d134751eaaabdb858fd23b08ffc1ef5fb270a753090299d86102c9a not found: ID does not exist"
Sep 16 21:44:18.430879 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.430859    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"]
Sep 16 21:44:18.436577 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.436550    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5ee44a89-openshift-default-868ff99448-rpc7s"]
Sep 16 21:44:18.609699 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:18.609603    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" path="/var/lib/kubelet/pods/6434fe14-9fe2-40ee-98f8-cc96be3d2fb9/volumes"
Sep 16 21:44:30.783760 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.783723    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"]
Sep 16 21:44:30.784236 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.784189    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" containerName="istio-proxy"
Sep 16 21:44:30.784236 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.784206    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" containerName="istio-proxy"
Sep 16 21:44:30.784236 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.784218    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" containerName="istio-proxy"
Sep 16 21:44:30.784236 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.784227    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" containerName="istio-proxy"
Sep 16 21:44:30.784456 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.784265    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3716ae16-a9c5-410d-821f-fde454c463db" containerName="main"
Sep 16 21:44:30.784456 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.784276    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3716ae16-a9c5-410d-821f-fde454c463db" containerName="main"
Sep 16 21:44:30.784456 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.784367    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" containerName="istio-proxy"
Sep 16 21:44:30.784456 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.784379    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6434fe14-9fe2-40ee-98f8-cc96be3d2fb9" containerName="istio-proxy"
Sep 16 21:44:30.784456 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.784387    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3716ae16-a9c5-410d-821f-fde454c463db" containerName="main"
Sep 16 21:44:30.788864 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.788845    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.792207 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.792176    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-35f5fd\"/\"embedding-w0-35f5fd-kserve-self-signed-certs\""
Sep 16 21:44:30.805914 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.805885    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"]
Sep 16 21:44:30.847439 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.847407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-home\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.847585 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.847469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-dshm\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.847585 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.847521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft7bz\" (UniqueName: \"kubernetes.io/projected/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-kube-api-access-ft7bz\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.847585 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.847538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-tls-certs\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.847585 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.847556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-tmp-dir\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.847771 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.847587    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-model-cache\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.948468 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.948437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-dshm\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.948645 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.948476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7bz\" (UniqueName: \"kubernetes.io/projected/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-kube-api-access-ft7bz\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.948645 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.948505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-tls-certs\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.948645 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.948524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-tmp-dir\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.948645 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.948554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-model-cache\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.948645 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.948617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-home\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.948991 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.948961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-tmp-dir\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.949110 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.948970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-home\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.949110 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.949041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-model-cache\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.950762 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.950737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-dshm\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.951025 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.951009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-tls-certs\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:30.956413 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:30.956391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft7bz\" (UniqueName: \"kubernetes.io/projected/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-kube-api-access-ft7bz\") pod \"embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") " pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:31.099576 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:31.099482    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:31.387587 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:31.387562    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"]
Sep 16 21:44:31.392739 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:44:31.392706    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2429b0cb_93a4_4a8c_99a2_15aa7f127ada.slice/crio-210046d68e461deaff07eeb7df6a9770256d7831649807040a5c27ec9c8149e1 WatchSource:0}: Error finding container 210046d68e461deaff07eeb7df6a9770256d7831649807040a5c27ec9c8149e1: Status 404 returned error can't find the container with id 210046d68e461deaff07eeb7df6a9770256d7831649807040a5c27ec9c8149e1
Sep 16 21:44:31.483483 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:31.483400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq" event={"ID":"2429b0cb-93a4-4a8c-99a2-15aa7f127ada","Type":"ContainerStarted","Data":"535f8c5f25b7df6a508b688f866443be31dc7eb97e58e5865d54317505a56341"}
Sep 16 21:44:31.483483 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:31.483458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq" event={"ID":"2429b0cb-93a4-4a8c-99a2-15aa7f127ada","Type":"ContainerStarted","Data":"210046d68e461deaff07eeb7df6a9770256d7831649807040a5c27ec9c8149e1"}
Sep 16 21:44:41.100141 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:41.100109    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:41.100560 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:41.100244    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:41.114370 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:41.114337    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:44:41.133698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:41.133638    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq" podStartSLOduration=11.133623981 podStartE2EDuration="11.133623981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:44:31.500490915 +0000 UTC m=+1331.400700922" watchObservedRunningTime="2026-09-16 21:44:41.133623981 +0000 UTC m=+1341.033833984"
Sep 16 21:44:41.546279 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:44:41.546246    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:45:00.135847 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.135759    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-35f5fd-29826585-4zc4f"]
Sep 16 21:45:00.139471 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.139449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-35f5fd-29826585-4zc4f"
Sep 16 21:45:00.141806 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.141783    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:45:00.142711 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.142692    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826585-njznj"]
Sep 16 21:45:00.146099 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.146076    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7df81-29826585-2bk6d"]
Sep 16 21:45:00.146889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.146159    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826585-njznj"
Sep 16 21:45:00.149960 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.149938    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-35f5fd-29826585-4zc4f"]
Sep 16 21:45:00.150156 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.150035    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7df81-29826585-2bk6d"
Sep 16 21:45:00.156614 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.154853    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826585-njznj"]
Sep 16 21:45:00.158420 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.158400    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-35f5fd-29826585-4zc4f"
Sep 16 21:45:00.165201 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.164168    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7df81-29826585-2bk6d"]
Sep 16 21:45:00.170455 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.170430    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e047d8-29826585-n8pjr"]
Sep 16 21:45:00.170586 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.170432    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826585-njznj"
Sep 16 21:45:00.174434 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.174413    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e047d8-29826585-n8pjr"
Sep 16 21:45:00.185160 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.185133    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e047d8-29826585-n8pjr"]
Sep 16 21:45:00.206016 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.205867    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7df81-29826585-2bk6d"
Sep 16 21:45:00.216673 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.215861    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e047d8-29826585-n8pjr"
Sep 16 21:45:00.455917 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.455887    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-35f5fd-29826585-4zc4f"]
Sep 16 21:45:00.459045 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:45:00.459018    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4489100_4917_4f1c_af70_7d8f53c1d833.slice/crio-6b54ab9bb553437d167c60f9d90b2b4b7161e8b3aee2006a0f0d078a2a09bba8 WatchSource:0}: Error finding container 6b54ab9bb553437d167c60f9d90b2b4b7161e8b3aee2006a0f0d078a2a09bba8: Status 404 returned error can't find the container with id 6b54ab9bb553437d167c60f9d90b2b4b7161e8b3aee2006a0f0d078a2a09bba8
Sep 16 21:45:00.468076 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.468053    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826585-njznj"]
Sep 16 21:45:00.471990 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:45:00.471968    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54596020_9528_40ab_8b32_bb8c9c5b7edb.slice/crio-86244cc730d1931dadf06552ba26b7bc46329dcc469b886bdf6b58993821aaf0 WatchSource:0}: Error finding container 86244cc730d1931dadf06552ba26b7bc46329dcc469b886bdf6b58993821aaf0: Status 404 returned error can't find the container with id 86244cc730d1931dadf06552ba26b7bc46329dcc469b886bdf6b58993821aaf0
Sep 16 21:45:00.521562 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.521533    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7df81-29826585-2bk6d"]
Sep 16 21:45:00.522924 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:45:00.522901    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa28239f_af43_4194_b456_a662576fa0b7.slice/crio-4c4da8c4be9ed9a636a31149bd14f548af317debfcc0241417c4668875a446ff WatchSource:0}: Error finding container 4c4da8c4be9ed9a636a31149bd14f548af317debfcc0241417c4668875a446ff: Status 404 returned error can't find the container with id 4c4da8c4be9ed9a636a31149bd14f548af317debfcc0241417c4668875a446ff
Sep 16 21:45:00.533041 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.533022    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e047d8-29826585-n8pjr"]
Sep 16 21:45:00.536239 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:45:00.536220    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ae327a_e0f1_4e4a_8303_a4d549a1fc8a.slice/crio-ed36d0fc0d7265c16b8e8d7f875e1f01731c1d70b8deb7a53ca1269a1d2525be WatchSource:0}: Error finding container ed36d0fc0d7265c16b8e8d7f875e1f01731c1d70b8deb7a53ca1269a1d2525be: Status 404 returned error can't find the container with id ed36d0fc0d7265c16b8e8d7f875e1f01731c1d70b8deb7a53ca1269a1d2525be
Sep 16 21:45:00.633897 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.633864    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e047d8-29826585-n8pjr" event={"ID":"25ae327a-e0f1-4e4a-8303-a4d549a1fc8a","Type":"ContainerStarted","Data":"ed36d0fc0d7265c16b8e8d7f875e1f01731c1d70b8deb7a53ca1269a1d2525be"}
Sep 16 21:45:00.635156 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.635134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-35f5fd-29826585-4zc4f" event={"ID":"b4489100-4917-4f1c-af70-7d8f53c1d833","Type":"ContainerStarted","Data":"6b54ab9bb553437d167c60f9d90b2b4b7161e8b3aee2006a0f0d078a2a09bba8"}
Sep 16 21:45:00.636292 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.636273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826585-njznj" event={"ID":"54596020-9528-40ab-8b32-bb8c9c5b7edb","Type":"ContainerStarted","Data":"86244cc730d1931dadf06552ba26b7bc46329dcc469b886bdf6b58993821aaf0"}
Sep 16 21:45:00.637399 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:00.637378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7df81-29826585-2bk6d" event={"ID":"fa28239f-af43-4194-b456-a662576fa0b7","Type":"ContainerStarted","Data":"4c4da8c4be9ed9a636a31149bd14f548af317debfcc0241417c4668875a446ff"}
Sep 16 21:45:01.644263 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:01.644226    2563 generic.go:356] "Generic (PLEG): container finished" podID="25ae327a-e0f1-4e4a-8303-a4d549a1fc8a" containerID="bd05e6e5be70d4fe040f18dac6a23efb9d9a584a94ddb28fe3efe8301b6a1cb0" exitCode=0
Sep 16 21:45:01.644676 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:01.644284    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e047d8-29826585-n8pjr" event={"ID":"25ae327a-e0f1-4e4a-8303-a4d549a1fc8a","Type":"ContainerDied","Data":"bd05e6e5be70d4fe040f18dac6a23efb9d9a584a94ddb28fe3efe8301b6a1cb0"}
Sep 16 21:45:01.645902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:01.645873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-35f5fd-29826585-4zc4f" event={"ID":"b4489100-4917-4f1c-af70-7d8f53c1d833","Type":"ContainerStarted","Data":"788fd3d50bc2aa87041456e65e7f0be1fb7dee30caebe97f56c685272581f679"}
Sep 16 21:45:01.647347 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:01.647318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826585-njznj" event={"ID":"54596020-9528-40ab-8b32-bb8c9c5b7edb","Type":"ContainerStarted","Data":"fe29757483abf0754da39897f45d298d32d6328464754805a5f61328bc4392d6"}
Sep 16 21:45:01.648938 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:01.648917    2563 generic.go:356] "Generic (PLEG): container finished" podID="fa28239f-af43-4194-b456-a662576fa0b7" containerID="67ddeead074ea1b8a74880941ec0d70eb302df0136d8a41d9ee381e3df7db14b" exitCode=0
Sep 16 21:45:01.649037 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:01.648943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7df81-29826585-2bk6d" event={"ID":"fa28239f-af43-4194-b456-a662576fa0b7","Type":"ContainerDied","Data":"67ddeead074ea1b8a74880941ec0d70eb302df0136d8a41d9ee381e3df7db14b"}
Sep 16 21:45:01.674119 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:01.674071    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826585-njznj" podStartSLOduration=1.6740533480000002 podStartE2EDuration="1.674053348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:45:01.671385739 +0000 UTC m=+1361.571595738" watchObservedRunningTime="2026-09-16 21:45:01.674053348 +0000 UTC m=+1361.574263353"
Sep 16 21:45:01.701598 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:01.701547    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-35f5fd-29826585-4zc4f" podStartSLOduration=1.701531444 podStartE2EDuration="1.701531444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:45:01.699800901 +0000 UTC m=+1361.600010898" watchObservedRunningTime="2026-09-16 21:45:01.701531444 +0000 UTC m=+1361.601741449"
Sep 16 21:45:02.656052 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:02.656018    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4489100-4917-4f1c-af70-7d8f53c1d833" containerID="788fd3d50bc2aa87041456e65e7f0be1fb7dee30caebe97f56c685272581f679" exitCode=0
Sep 16 21:45:02.656473 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:02.656113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-35f5fd-29826585-4zc4f" event={"ID":"b4489100-4917-4f1c-af70-7d8f53c1d833","Type":"ContainerDied","Data":"788fd3d50bc2aa87041456e65e7f0be1fb7dee30caebe97f56c685272581f679"}
Sep 16 21:45:02.657710 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:02.657688    2563 generic.go:356] "Generic (PLEG): container finished" podID="54596020-9528-40ab-8b32-bb8c9c5b7edb" containerID="fe29757483abf0754da39897f45d298d32d6328464754805a5f61328bc4392d6" exitCode=0
Sep 16 21:45:02.657808 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:02.657772    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826585-njznj" event={"ID":"54596020-9528-40ab-8b32-bb8c9c5b7edb","Type":"ContainerDied","Data":"fe29757483abf0754da39897f45d298d32d6328464754805a5f61328bc4392d6"}
Sep 16 21:45:02.827981 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:02.827958    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e047d8-29826585-n8pjr"
Sep 16 21:45:02.830766 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:02.830747    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7df81-29826585-2bk6d"
Sep 16 21:45:03.664644 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:03.664608    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e047d8-29826585-n8pjr"
Sep 16 21:45:03.665116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:03.664638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e047d8-29826585-n8pjr" event={"ID":"25ae327a-e0f1-4e4a-8303-a4d549a1fc8a","Type":"ContainerDied","Data":"ed36d0fc0d7265c16b8e8d7f875e1f01731c1d70b8deb7a53ca1269a1d2525be"}
Sep 16 21:45:03.665116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:03.664719    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed36d0fc0d7265c16b8e8d7f875e1f01731c1d70b8deb7a53ca1269a1d2525be"
Sep 16 21:45:03.666020 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:03.665997    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7df81-29826585-2bk6d"
Sep 16 21:45:03.666092 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:03.666038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7df81-29826585-2bk6d" event={"ID":"fa28239f-af43-4194-b456-a662576fa0b7","Type":"ContainerDied","Data":"4c4da8c4be9ed9a636a31149bd14f548af317debfcc0241417c4668875a446ff"}
Sep 16 21:45:03.666092 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:03.666078    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c4da8c4be9ed9a636a31149bd14f548af317debfcc0241417c4668875a446ff"
Sep 16 21:45:03.817499 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:03.817471    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-35f5fd-29826585-4zc4f"
Sep 16 21:45:03.820427 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:03.820406    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826585-njznj"
Sep 16 21:45:04.672910 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:04.672879    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-35f5fd-29826585-4zc4f"
Sep 16 21:45:04.673355 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:04.672878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-35f5fd-29826585-4zc4f" event={"ID":"b4489100-4917-4f1c-af70-7d8f53c1d833","Type":"ContainerDied","Data":"6b54ab9bb553437d167c60f9d90b2b4b7161e8b3aee2006a0f0d078a2a09bba8"}
Sep 16 21:45:04.673355 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:04.673004    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b54ab9bb553437d167c60f9d90b2b4b7161e8b3aee2006a0f0d078a2a09bba8"
Sep 16 21:45:04.674301 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:04.674277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826585-njznj" event={"ID":"54596020-9528-40ab-8b32-bb8c9c5b7edb","Type":"ContainerDied","Data":"86244cc730d1931dadf06552ba26b7bc46329dcc469b886bdf6b58993821aaf0"}
Sep 16 21:45:04.674301 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:04.674299    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826585-njznj"
Sep 16 21:45:04.674460 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:04.674314    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86244cc730d1931dadf06552ba26b7bc46329dcc469b886bdf6b58993821aaf0"
Sep 16 21:45:07.468992 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.468959    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"]
Sep 16 21:45:07.469577 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.469559    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa28239f-af43-4194-b456-a662576fa0b7" containerName="cleanup"
Sep 16 21:45:07.469626 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.469580    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa28239f-af43-4194-b456-a662576fa0b7" containerName="cleanup"
Sep 16 21:45:07.469626 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.469611    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25ae327a-e0f1-4e4a-8303-a4d549a1fc8a" containerName="cleanup"
Sep 16 21:45:07.469626 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.469620    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25ae327a-e0f1-4e4a-8303-a4d549a1fc8a" containerName="cleanup"
Sep 16 21:45:07.469741 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.469642    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54596020-9528-40ab-8b32-bb8c9c5b7edb" containerName="cleanup"
Sep 16 21:45:07.469741 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.469679    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="54596020-9528-40ab-8b32-bb8c9c5b7edb" containerName="cleanup"
Sep 16 21:45:07.469741 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.469703    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4489100-4917-4f1c-af70-7d8f53c1d833" containerName="cleanup"
Sep 16 21:45:07.469741 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.469712    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4489100-4917-4f1c-af70-7d8f53c1d833" containerName="cleanup"
Sep 16 21:45:07.469864 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.469818    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa28239f-af43-4194-b456-a662576fa0b7" containerName="cleanup"
Sep 16 21:45:07.469864 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.469832    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="54596020-9528-40ab-8b32-bb8c9c5b7edb" containerName="cleanup"
Sep 16 21:45:07.469864 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.469849    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4489100-4917-4f1c-af70-7d8f53c1d833" containerName="cleanup"
Sep 16 21:45:07.469864 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.469859    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="25ae327a-e0f1-4e4a-8303-a4d549a1fc8a" containerName="cleanup"
Sep 16 21:45:07.473038 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.473019    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.475445 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.475424    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0de41b-openshift-default-dockercfg-98mgj\""
Sep 16 21:45:07.485940 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.485914    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"]
Sep 16 21:45:07.544365 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.544330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84974a56-ec81-435f-a8af-207b51fadedb-istio-token\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.544541 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.544368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84974a56-ec81-435f-a8af-207b51fadedb-istiod-ca-cert\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.544541 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.544392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4s6z\" (UniqueName: \"kubernetes.io/projected/84974a56-ec81-435f-a8af-207b51fadedb-kube-api-access-p4s6z\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.544541 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.544476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-istio-data\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.544541 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.544514    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-workload-certs\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.544714 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.544635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-istio-envoy\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.544714 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.544684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84974a56-ec81-435f-a8af-207b51fadedb-istio-podinfo\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.544783 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.544715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-credential-socket\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.544783 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.544744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-workload-socket\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.645673 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.645624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-workload-certs\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.645875 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.645734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-istio-envoy\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.645875 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.645757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84974a56-ec81-435f-a8af-207b51fadedb-istio-podinfo\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.645875 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.645776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-credential-socket\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.645875 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.645802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-workload-socket\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.645875 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.645866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84974a56-ec81-435f-a8af-207b51fadedb-istio-token\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.646156 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.645889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84974a56-ec81-435f-a8af-207b51fadedb-istiod-ca-cert\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.646156 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.645911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4s6z\" (UniqueName: \"kubernetes.io/projected/84974a56-ec81-435f-a8af-207b51fadedb-kube-api-access-p4s6z\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.646156 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.645936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-istio-data\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.646156 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.646077    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-workload-certs\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.646352 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.646176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-credential-socket\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.646352 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.646314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-istio-data\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.646440 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.646425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-workload-socket\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.646733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.646709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84974a56-ec81-435f-a8af-207b51fadedb-istiod-ca-cert\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.648104 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.648080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-istio-envoy\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.648312 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.648296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84974a56-ec81-435f-a8af-207b51fadedb-istio-podinfo\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.653456 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.653431    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4s6z\" (UniqueName: \"kubernetes.io/projected/84974a56-ec81-435f-a8af-207b51fadedb-kube-api-access-p4s6z\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.653814 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.653797    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84974a56-ec81-435f-a8af-207b51fadedb-istio-token\") pod \"e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") " pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:07.784228 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:07.784133    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:08.059604 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:08.059575    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"]
Sep 16 21:45:08.061615 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:45:08.061587    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84974a56_ec81_435f_a8af_207b51fadedb.slice/crio-f31c7118f7cbc594b8caff558ca27766c9ea3ee9f41e8eacb6eda6a8524f3542 WatchSource:0}: Error finding container f31c7118f7cbc594b8caff558ca27766c9ea3ee9f41e8eacb6eda6a8524f3542: Status 404 returned error can't find the container with id f31c7118f7cbc594b8caff558ca27766c9ea3ee9f41e8eacb6eda6a8524f3542
Sep 16 21:45:08.064402 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:08.064366    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:45:08.064518 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:08.064450    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:45:08.064518 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:08.064494    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:45:08.704970 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:08.704928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" event={"ID":"84974a56-ec81-435f-a8af-207b51fadedb","Type":"ContainerStarted","Data":"f2f22226b91362faeee63e7ae083fc763296d3f8eaf5607b5e40fb6be7567a5f"}
Sep 16 21:45:08.705552 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:08.705525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" event={"ID":"84974a56-ec81-435f-a8af-207b51fadedb","Type":"ContainerStarted","Data":"f31c7118f7cbc594b8caff558ca27766c9ea3ee9f41e8eacb6eda6a8524f3542"}
Sep 16 21:45:08.743001 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:08.742919    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" podStartSLOduration=1.742892356 podStartE2EDuration="1.742892356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:45:08.737639376 +0000 UTC m=+1368.637849385" watchObservedRunningTime="2026-09-16 21:45:08.742892356 +0000 UTC m=+1368.643102361"
Sep 16 21:45:08.784847 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:08.784804    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:09.789908 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:09.789880    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:10.716412 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:10.716376    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:10.717454 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:10.717428    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:20.829778 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:20.829747    2563 scope.go:117] "RemoveContainer" containerID="ea98e5f1dfa6bda4a01f49cf896a7b7c0b7f6a79dfd084b7d227555c95d6f804"
Sep 16 21:45:39.377566 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:39.377528    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44"]
Sep 16 21:45:39.382802 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:39.382783    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44"
Sep 16 21:45:39.387019 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:39.386244    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:45:39.387019 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:39.386379    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:45:39.387019 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:39.386953    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44"]
Sep 16 21:45:39.495384 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:39.495349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44\" (UID: \"0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44"
Sep 16 21:45:39.596854 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:39.596818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44\" (UID: \"0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44"
Sep 16 21:45:39.597429 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:39.597408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44\" (UID: \"0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44"
Sep 16 21:45:39.697194 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:39.697158    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44"
Sep 16 21:45:39.975704 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:39.975671    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44"]
Sep 16 21:45:39.977095 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:45:39.977055    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8ea861_aa97_4b5c_88de_cc29d1f7b5f7.slice/crio-f2dbe0501e07d46824c12d4e1e64536fcd998797859686b704ea054fb60afcdc WatchSource:0}: Error finding container f2dbe0501e07d46824c12d4e1e64536fcd998797859686b704ea054fb60afcdc: Status 404 returned error can't find the container with id f2dbe0501e07d46824c12d4e1e64536fcd998797859686b704ea054fb60afcdc
Sep 16 21:45:40.887593 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:40.887559    2563 generic.go:356] "Generic (PLEG): container finished" podID="0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7" containerID="742c84b5e5adf02c57171325512df79b1c7c3b17ba2777de11f209aabc985914" exitCode=0
Sep 16 21:45:40.887980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:40.887645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44" event={"ID":"0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7","Type":"ContainerDied","Data":"742c84b5e5adf02c57171325512df79b1c7c3b17ba2777de11f209aabc985914"}
Sep 16 21:45:40.887980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:40.887713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44" event={"ID":"0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7","Type":"ContainerStarted","Data":"f2dbe0501e07d46824c12d4e1e64536fcd998797859686b704ea054fb60afcdc"}
Sep 16 21:45:41.243940 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.243038    2563 patch_prober.go:28] interesting pod/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": read tcp 10.133.0.2:38926->10.133.0.72:15021: read: connection reset by peer" start-of-body=
Sep 16 21:45:41.243940 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.243105    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" podUID="84974a56-ec81-435f-a8af-207b51fadedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": read tcp 10.133.0.2:38926->10.133.0.72:15021: read: connection reset by peer"
Sep 16 21:45:41.492980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.492890    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"]
Sep 16 21:45:41.496499 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.496474    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.499359 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.499333    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0102fa-openshift-default-dockercfg-l2mtc\""
Sep 16 21:45:41.511104 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.511076    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"]
Sep 16 21:45:41.621220 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.621068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-podinfo\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.621220 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.621129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-workload-socket\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.621220 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.621162    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-credential-socket\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.621220 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.621186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-workload-certs\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.621628 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.621320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-token\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.621628 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.621404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-envoy\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.621628 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.621494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djcmh\" (UniqueName: \"kubernetes.io/projected/11ca57ee-cf8a-41a5-9769-cc43a25f20af-kube-api-access-djcmh\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.621628 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.621528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-data\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.621628 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.621565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istiod-ca-cert\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.722626 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.722592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-podinfo\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.722626 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.722630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-workload-socket\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.722870 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.722649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-credential-socket\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.722870 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.722810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-workload-certs\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.722965 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.722888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-token\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.723040 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.723016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-envoy\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.723185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.723043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-credential-socket\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.723185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.723092    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-workload-socket\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.723185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.723129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djcmh\" (UniqueName: \"kubernetes.io/projected/11ca57ee-cf8a-41a5-9769-cc43a25f20af-kube-api-access-djcmh\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.723185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.723150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-workload-certs\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.723185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.723169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-data\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.723463 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.723218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istiod-ca-cert\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.723527 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.723478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-data\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.723876 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.723855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istiod-ca-cert\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.725187 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.725163    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-envoy\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.725345 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.725331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-podinfo\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.730828 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.730802    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-token\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.730828 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.730822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djcmh\" (UniqueName: \"kubernetes.io/projected/11ca57ee-cf8a-41a5-9769-cc43a25f20af-kube-api-access-djcmh\") pod \"e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") " pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.807302 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.807215    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:41.903948 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.903909    2563 generic.go:356] "Generic (PLEG): container finished" podID="84974a56-ec81-435f-a8af-207b51fadedb" containerID="f2f22226b91362faeee63e7ae083fc763296d3f8eaf5607b5e40fb6be7567a5f" exitCode=0
Sep 16 21:45:41.904421 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.904014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" event={"ID":"84974a56-ec81-435f-a8af-207b51fadedb","Type":"ContainerDied","Data":"f2f22226b91362faeee63e7ae083fc763296d3f8eaf5607b5e40fb6be7567a5f"}
Sep 16 21:45:41.904876 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.904789    2563 scope.go:117] "RemoveContainer" containerID="f2f22226b91362faeee63e7ae083fc763296d3f8eaf5607b5e40fb6be7567a5f"
Sep 16 21:45:41.906237 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.906198    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:45:41.906331 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.906267    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:45:41.906331 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:41.906302    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:45:42.035130 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:42.035102    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44"
Sep 16 21:45:42.084296 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:42.084267    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"]
Sep 16 21:45:42.084765 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:45:42.084737    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ca57ee_cf8a_41a5_9769_cc43a25f20af.slice/crio-c985bde1566eaa5dc1518f67fd443ef87c0c2de67aa00fb8eee1a99cf36e7747 WatchSource:0}: Error finding container c985bde1566eaa5dc1518f67fd443ef87c0c2de67aa00fb8eee1a99cf36e7747: Status 404 returned error can't find the container with id c985bde1566eaa5dc1518f67fd443ef87c0c2de67aa00fb8eee1a99cf36e7747
Sep 16 21:45:42.087814 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:42.087778    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:45:42.087911 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:42.087848    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:45:42.087911 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:42.087882    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:45:42.126793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:42.126766    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7-maas-api-service-ca\") pod \"0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7\" (UID: \"0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7\") "
Sep 16 21:45:42.127386 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:42.127143    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7" (UID: "0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:45:42.228351 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:42.228312    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7-maas-api-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:42.911091 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:42.911004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" event={"ID":"84974a56-ec81-435f-a8af-207b51fadedb","Type":"ContainerStarted","Data":"e559f1c9b8b4f80076093976c113da97dc8e62ae0671df84f883898421fa2fdb"}
Sep 16 21:45:42.912710 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:42.912683    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8" event={"ID":"11ca57ee-cf8a-41a5-9769-cc43a25f20af","Type":"ContainerStarted","Data":"a42847133c9dde080d49326f99275c0c7f37420fb8c1318c27b8e3a237cc0e7d"}
Sep 16 21:45:42.912832 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:42.912725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8" event={"ID":"11ca57ee-cf8a-41a5-9769-cc43a25f20af","Type":"ContainerStarted","Data":"c985bde1566eaa5dc1518f67fd443ef87c0c2de67aa00fb8eee1a99cf36e7747"}
Sep 16 21:45:42.914027 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:42.914010    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44"
Sep 16 21:45:42.914100 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:42.914055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44" event={"ID":"0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7","Type":"ContainerDied","Data":"f2dbe0501e07d46824c12d4e1e64536fcd998797859686b704ea054fb60afcdc"}
Sep 16 21:45:42.914100 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:42.914093    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2dbe0501e07d46824c12d4e1e64536fcd998797859686b704ea054fb60afcdc"
Sep 16 21:45:42.953101 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:42.953056    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8" podStartSLOduration=1.953040837 podStartE2EDuration="1.953040837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:45:42.951692851 +0000 UTC m=+1402.851902855" watchObservedRunningTime="2026-09-16 21:45:42.953040837 +0000 UTC m=+1402.853250841"
Sep 16 21:45:43.785337 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:43.785303    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:43.785547 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:43.785509    2563 patch_prober.go:28] interesting pod/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 16 21:45:43.785602 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:43.785574    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" podUID="84974a56-ec81-435f-a8af-207b51fadedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 16 21:45:43.808251 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:43.808225    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:43.813544 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:43.813514    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:43.920351 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:43.920312    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:43.921245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:43.921224    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:45:44.218579 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:44.218547    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"]
Sep 16 21:45:44.234975 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:44.234946    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"]
Sep 16 21:45:44.248948 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:44.248921    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"]
Sep 16 21:45:44.785146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:44.785108    2563 patch_prober.go:28] interesting pod/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 16 21:45:44.785914 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:44.785442    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" podUID="84974a56-ec81-435f-a8af-207b51fadedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 16 21:45:45.785242 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:45.785192    2563 patch_prober.go:28] interesting pod/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 16 21:45:45.785802 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:45.785282    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" podUID="84974a56-ec81-435f-a8af-207b51fadedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 16 21:45:46.785003 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:46.784972    2563 patch_prober.go:28] interesting pod/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 16 21:45:46.785299 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:46.785270    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" podUID="84974a56-ec81-435f-a8af-207b51fadedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 16 21:45:47.785369 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:47.785331    2563 patch_prober.go:28] interesting pod/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 16 21:45:47.786012 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:47.785978    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" podUID="84974a56-ec81-435f-a8af-207b51fadedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 16 21:45:48.785155 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:48.785102    2563 patch_prober.go:28] interesting pod/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 16 21:45:48.785355 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:48.785178    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" podUID="84974a56-ec81-435f-a8af-207b51fadedb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 16 21:45:49.854695 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:49.854634    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:49.855180 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:49.855048    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:49.857176 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:49.857155    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:45:58.846247 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:58.846211    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-35f5fd-29826585-4zc4f"]
Sep 16 21:45:58.848632 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:58.848606    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-35f5fd-29826585-4zc4f"]
Sep 16 21:45:59.042726 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.042689    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44"]
Sep 16 21:45:59.045044 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.045019    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-35f5fd-kjl44"]
Sep 16 21:45:59.241580 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.241518    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28" podUID="bdaf67ad-cce8-4dfe-9f82-48df7fcaed88" containerName="main" containerID="cri-o://25d20f95e1a8f6d6c11a3ad833b520bbb863ad08fe1d3c9d48b7647edcbe634d" gracePeriod=45
Sep 16 21:45:59.257522 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.257321    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq" podUID="2429b0cb-93a4-4a8c-99a2-15aa7f127ada" containerName="main" containerID="cri-o://535f8c5f25b7df6a508b688f866443be31dc7eb97e58e5865d54317505a56341" gracePeriod=45
Sep 16 21:45:59.276958 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.276923    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d" podUID="bd82ee24-7331-4ced-9aaa-80b6a3739348" containerName="main" containerID="cri-o://335ffaee820105652f2f0a879804f9dc4831806589f014c328265508e4339f6e" gracePeriod=45
Sep 16 21:45:59.542064 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.542027    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:45:59.590080 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.590040    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:45:59.592173 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.592151    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:45:59.628311 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628270    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwx46\" (UniqueName: \"kubernetes.io/projected/bd82ee24-7331-4ced-9aaa-80b6a3739348-kube-api-access-kwx46\") pod \"bd82ee24-7331-4ced-9aaa-80b6a3739348\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") "
Sep 16 21:45:59.628473 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628322    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-model-cache\") pod \"bd82ee24-7331-4ced-9aaa-80b6a3739348\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") "
Sep 16 21:45:59.628473 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628366    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-dshm\") pod \"bd82ee24-7331-4ced-9aaa-80b6a3739348\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") "
Sep 16 21:45:59.628473 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628424    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-dshm\") pod \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") "
Sep 16 21:45:59.628473 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628448    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ft7bz\" (UniqueName: \"kubernetes.io/projected/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-kube-api-access-ft7bz\") pod \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") "
Sep 16 21:45:59.628727 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628484    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-model-cache\") pod \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") "
Sep 16 21:45:59.628727 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628530    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-tmp-dir\") pod \"bd82ee24-7331-4ced-9aaa-80b6a3739348\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") "
Sep 16 21:45:59.628727 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628557    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-tls-certs\") pod \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") "
Sep 16 21:45:59.628727 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628580    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-home\") pod \"bd82ee24-7331-4ced-9aaa-80b6a3739348\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") "
Sep 16 21:45:59.628727 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628627    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-dshm\") pod \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") "
Sep 16 21:45:59.628727 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628630    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-model-cache" (OuterVolumeSpecName: "model-cache") pod "bd82ee24-7331-4ced-9aaa-80b6a3739348" (UID: "bd82ee24-7331-4ced-9aaa-80b6a3739348"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.629033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628730    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd82ee24-7331-4ced-9aaa-80b6a3739348-tls-certs\") pod \"bd82ee24-7331-4ced-9aaa-80b6a3739348\" (UID: \"bd82ee24-7331-4ced-9aaa-80b6a3739348\") "
Sep 16 21:45:59.629033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628766    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-tmp-dir\") pod \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") "
Sep 16 21:45:59.629033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628797    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-model-cache\") pod \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") "
Sep 16 21:45:59.629033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628829    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g2l5\" (UniqueName: \"kubernetes.io/projected/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-kube-api-access-7g2l5\") pod \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") "
Sep 16 21:45:59.629033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628854    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-home\") pod \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") "
Sep 16 21:45:59.629033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628884    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-home\") pod \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") "
Sep 16 21:45:59.629033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.628929    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-tls-certs\") pod \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\" (UID: \"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88\") "
Sep 16 21:45:59.629033 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.629009    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-tmp-dir\") pod \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\" (UID: \"2429b0cb-93a4-4a8c-99a2-15aa7f127ada\") "
Sep 16 21:45:59.629820 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.629796    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-model-cache\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.630296 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.630272    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2429b0cb-93a4-4a8c-99a2-15aa7f127ada" (UID: "2429b0cb-93a4-4a8c-99a2-15aa7f127ada"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.630923 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.630900    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-model-cache" (OuterVolumeSpecName: "model-cache") pod "2429b0cb-93a4-4a8c-99a2-15aa7f127ada" (UID: "2429b0cb-93a4-4a8c-99a2-15aa7f127ada"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.631028 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.631003    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdaf67ad-cce8-4dfe-9f82-48df7fcaed88" (UID: "bdaf67ad-cce8-4dfe-9f82-48df7fcaed88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.631459 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.631404    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bd82ee24-7331-4ced-9aaa-80b6a3739348" (UID: "bd82ee24-7331-4ced-9aaa-80b6a3739348"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.631550 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.631514    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-dshm" (OuterVolumeSpecName: "dshm") pod "bdaf67ad-cce8-4dfe-9f82-48df7fcaed88" (UID: "bdaf67ad-cce8-4dfe-9f82-48df7fcaed88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.631714 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.631691    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-home" (OuterVolumeSpecName: "home") pod "2429b0cb-93a4-4a8c-99a2-15aa7f127ada" (UID: "2429b0cb-93a4-4a8c-99a2-15aa7f127ada"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.632012 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.631953    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdaf67ad-cce8-4dfe-9f82-48df7fcaed88" (UID: "bdaf67ad-cce8-4dfe-9f82-48df7fcaed88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.632166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.632146    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-home" (OuterVolumeSpecName: "home") pod "bd82ee24-7331-4ced-9aaa-80b6a3739348" (UID: "bd82ee24-7331-4ced-9aaa-80b6a3739348"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.632262 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.632221    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd82ee24-7331-4ced-9aaa-80b6a3739348-kube-api-access-kwx46" (OuterVolumeSpecName: "kube-api-access-kwx46") pod "bd82ee24-7331-4ced-9aaa-80b6a3739348" (UID: "bd82ee24-7331-4ced-9aaa-80b6a3739348"). InnerVolumeSpecName "kube-api-access-kwx46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:45:59.632327 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.632238    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-home" (OuterVolumeSpecName: "home") pod "bdaf67ad-cce8-4dfe-9f82-48df7fcaed88" (UID: "bdaf67ad-cce8-4dfe-9f82-48df7fcaed88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.632850 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.632823    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-dshm" (OuterVolumeSpecName: "dshm") pod "bd82ee24-7331-4ced-9aaa-80b6a3739348" (UID: "bd82ee24-7331-4ced-9aaa-80b6a3739348"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.633362 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.633336    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-dshm" (OuterVolumeSpecName: "dshm") pod "2429b0cb-93a4-4a8c-99a2-15aa7f127ada" (UID: "2429b0cb-93a4-4a8c-99a2-15aa7f127ada"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.633739 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.633714    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd82ee24-7331-4ced-9aaa-80b6a3739348-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bd82ee24-7331-4ced-9aaa-80b6a3739348" (UID: "bd82ee24-7331-4ced-9aaa-80b6a3739348"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:45:59.634583 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.634284    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2429b0cb-93a4-4a8c-99a2-15aa7f127ada" (UID: "2429b0cb-93a4-4a8c-99a2-15aa7f127ada"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:45:59.634796 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.634771    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-kube-api-access-ft7bz" (OuterVolumeSpecName: "kube-api-access-ft7bz") pod "2429b0cb-93a4-4a8c-99a2-15aa7f127ada" (UID: "2429b0cb-93a4-4a8c-99a2-15aa7f127ada"). InnerVolumeSpecName "kube-api-access-ft7bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:45:59.635449 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.635424    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdaf67ad-cce8-4dfe-9f82-48df7fcaed88" (UID: "bdaf67ad-cce8-4dfe-9f82-48df7fcaed88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:45:59.635563 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.635536    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-kube-api-access-7g2l5" (OuterVolumeSpecName: "kube-api-access-7g2l5") pod "bdaf67ad-cce8-4dfe-9f82-48df7fcaed88" (UID: "bdaf67ad-cce8-4dfe-9f82-48df7fcaed88"). InnerVolumeSpecName "kube-api-access-7g2l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:45:59.730932 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.730892    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-tmp-dir\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.730932 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.730923    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-tls-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.730932 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.730935    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-home\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.731209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.730946    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-dshm\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.731209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.730956    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd82ee24-7331-4ced-9aaa-80b6a3739348-tls-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.731209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.730966    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-tmp-dir\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.731209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.730976    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-model-cache\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.731209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.730987    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g2l5\" (UniqueName: \"kubernetes.io/projected/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-kube-api-access-7g2l5\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.731209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.730997    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-home\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.731209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.731008    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-home\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.731209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.731018    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88-tls-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.731209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.731028    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-tmp-dir\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.731209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.731039    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwx46\" (UniqueName: \"kubernetes.io/projected/bd82ee24-7331-4ced-9aaa-80b6a3739348-kube-api-access-kwx46\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.731209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.731049    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd82ee24-7331-4ced-9aaa-80b6a3739348-dshm\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.731209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.731061    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-dshm\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.731209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.731072    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft7bz\" (UniqueName: \"kubernetes.io/projected/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-kube-api-access-ft7bz\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.731209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:45:59.731084    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2429b0cb-93a4-4a8c-99a2-15aa7f127ada-model-cache\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:00.015893 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.015858    2563 generic.go:356] "Generic (PLEG): container finished" podID="2429b0cb-93a4-4a8c-99a2-15aa7f127ada" containerID="535f8c5f25b7df6a508b688f866443be31dc7eb97e58e5865d54317505a56341" exitCode=0
Sep 16 21:46:00.016318 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.015916    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"
Sep 16 21:46:00.016318 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.015949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq" event={"ID":"2429b0cb-93a4-4a8c-99a2-15aa7f127ada","Type":"ContainerDied","Data":"535f8c5f25b7df6a508b688f866443be31dc7eb97e58e5865d54317505a56341"}
Sep 16 21:46:00.016318 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.016005    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq" event={"ID":"2429b0cb-93a4-4a8c-99a2-15aa7f127ada","Type":"ContainerDied","Data":"210046d68e461deaff07eeb7df6a9770256d7831649807040a5c27ec9c8149e1"}
Sep 16 21:46:00.016318 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.016028    2563 scope.go:117] "RemoveContainer" containerID="535f8c5f25b7df6a508b688f866443be31dc7eb97e58e5865d54317505a56341"
Sep 16 21:46:00.017592 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.017567    2563 generic.go:356] "Generic (PLEG): container finished" podID="bd82ee24-7331-4ced-9aaa-80b6a3739348" containerID="335ffaee820105652f2f0a879804f9dc4831806589f014c328265508e4339f6e" exitCode=0
Sep 16 21:46:00.017710 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.017623    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"
Sep 16 21:46:00.017710 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.017643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d" event={"ID":"bd82ee24-7331-4ced-9aaa-80b6a3739348","Type":"ContainerDied","Data":"335ffaee820105652f2f0a879804f9dc4831806589f014c328265508e4339f6e"}
Sep 16 21:46:00.017710 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.017704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d" event={"ID":"bd82ee24-7331-4ced-9aaa-80b6a3739348","Type":"ContainerDied","Data":"5b5c2edaec95e805e080c57066c807480135eed4090d2a61f263a49c46c0dd49"}
Sep 16 21:46:00.019566 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.019459    2563 generic.go:356] "Generic (PLEG): container finished" podID="bdaf67ad-cce8-4dfe-9f82-48df7fcaed88" containerID="25d20f95e1a8f6d6c11a3ad833b520bbb863ad08fe1d3c9d48b7647edcbe634d" exitCode=0
Sep 16 21:46:00.019566 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.019515    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"
Sep 16 21:46:00.019566 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.019524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28" event={"ID":"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88","Type":"ContainerDied","Data":"25d20f95e1a8f6d6c11a3ad833b520bbb863ad08fe1d3c9d48b7647edcbe634d"}
Sep 16 21:46:00.019810 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.019586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28" event={"ID":"bdaf67ad-cce8-4dfe-9f82-48df7fcaed88","Type":"ContainerDied","Data":"61ad5c7192324eb507f7e2284dd0e9a220249eeb51d26126a6f7b5b7a85176fb"}
Sep 16 21:46:00.027076 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.027053    2563 scope.go:117] "RemoveContainer" containerID="535f8c5f25b7df6a508b688f866443be31dc7eb97e58e5865d54317505a56341"
Sep 16 21:46:00.027833 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:46:00.027803    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535f8c5f25b7df6a508b688f866443be31dc7eb97e58e5865d54317505a56341\": container with ID starting with 535f8c5f25b7df6a508b688f866443be31dc7eb97e58e5865d54317505a56341 not found: ID does not exist" containerID="535f8c5f25b7df6a508b688f866443be31dc7eb97e58e5865d54317505a56341"
Sep 16 21:46:00.027948 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.027851    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535f8c5f25b7df6a508b688f866443be31dc7eb97e58e5865d54317505a56341"} err="failed to get container status \"535f8c5f25b7df6a508b688f866443be31dc7eb97e58e5865d54317505a56341\": rpc error: code = NotFound desc = could not find container \"535f8c5f25b7df6a508b688f866443be31dc7eb97e58e5865d54317505a56341\": container with ID starting with 535f8c5f25b7df6a508b688f866443be31dc7eb97e58e5865d54317505a56341 not found: ID does not exist"
Sep 16 21:46:00.027948 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.027868    2563 scope.go:117] "RemoveContainer" containerID="335ffaee820105652f2f0a879804f9dc4831806589f014c328265508e4339f6e"
Sep 16 21:46:00.037820 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.037796    2563 scope.go:117] "RemoveContainer" containerID="335ffaee820105652f2f0a879804f9dc4831806589f014c328265508e4339f6e"
Sep 16 21:46:00.038164 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:46:00.038139    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335ffaee820105652f2f0a879804f9dc4831806589f014c328265508e4339f6e\": container with ID starting with 335ffaee820105652f2f0a879804f9dc4831806589f014c328265508e4339f6e not found: ID does not exist" containerID="335ffaee820105652f2f0a879804f9dc4831806589f014c328265508e4339f6e"
Sep 16 21:46:00.038250 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.038171    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335ffaee820105652f2f0a879804f9dc4831806589f014c328265508e4339f6e"} err="failed to get container status \"335ffaee820105652f2f0a879804f9dc4831806589f014c328265508e4339f6e\": rpc error: code = NotFound desc = could not find container \"335ffaee820105652f2f0a879804f9dc4831806589f014c328265508e4339f6e\": container with ID starting with 335ffaee820105652f2f0a879804f9dc4831806589f014c328265508e4339f6e not found: ID does not exist"
Sep 16 21:46:00.038250 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.038190    2563 scope.go:117] "RemoveContainer" containerID="25d20f95e1a8f6d6c11a3ad833b520bbb863ad08fe1d3c9d48b7647edcbe634d"
Sep 16 21:46:00.043568 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.043547    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"]
Sep 16 21:46:00.048353 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.048326    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/embedding-w0-35f5fd-kserve-5b95dd8bd5-b5nqq"]
Sep 16 21:46:00.049018 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.049001    2563 scope.go:117] "RemoveContainer" containerID="25d20f95e1a8f6d6c11a3ad833b520bbb863ad08fe1d3c9d48b7647edcbe634d"
Sep 16 21:46:00.049330 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:46:00.049310    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d20f95e1a8f6d6c11a3ad833b520bbb863ad08fe1d3c9d48b7647edcbe634d\": container with ID starting with 25d20f95e1a8f6d6c11a3ad833b520bbb863ad08fe1d3c9d48b7647edcbe634d not found: ID does not exist" containerID="25d20f95e1a8f6d6c11a3ad833b520bbb863ad08fe1d3c9d48b7647edcbe634d"
Sep 16 21:46:00.049397 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.049341    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d20f95e1a8f6d6c11a3ad833b520bbb863ad08fe1d3c9d48b7647edcbe634d"} err="failed to get container status \"25d20f95e1a8f6d6c11a3ad833b520bbb863ad08fe1d3c9d48b7647edcbe634d\": rpc error: code = NotFound desc = could not find container \"25d20f95e1a8f6d6c11a3ad833b520bbb863ad08fe1d3c9d48b7647edcbe634d\": container with ID starting with 25d20f95e1a8f6d6c11a3ad833b520bbb863ad08fe1d3c9d48b7647edcbe634d not found: ID does not exist"
Sep 16 21:46:00.057595 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.057568    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"]
Sep 16 21:46:00.061002 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.060982    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/simulator-w0-35f5fd-kserve-75559497f6-6tf4d"]
Sep 16 21:46:00.070480 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.070459    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"]
Sep 16 21:46:00.077064 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.077043    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/distinct-2-w0-35f5fd-kserve-7c4cb4c47-fpb28"]
Sep 16 21:46:00.608571 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.608492    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7" path="/var/lib/kubelet/pods/0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7/volumes"
Sep 16 21:46:00.608928 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.608915    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2429b0cb-93a4-4a8c-99a2-15aa7f127ada" path="/var/lib/kubelet/pods/2429b0cb-93a4-4a8c-99a2-15aa7f127ada/volumes"
Sep 16 21:46:00.609170 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.609158    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4489100-4917-4f1c-af70-7d8f53c1d833" path="/var/lib/kubelet/pods/b4489100-4917-4f1c-af70-7d8f53c1d833/volumes"
Sep 16 21:46:00.609482 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.609470    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd82ee24-7331-4ced-9aaa-80b6a3739348" path="/var/lib/kubelet/pods/bd82ee24-7331-4ced-9aaa-80b6a3739348/volumes"
Sep 16 21:46:00.610158 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:00.610144    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdaf67ad-cce8-4dfe-9f82-48df7fcaed88" path="/var/lib/kubelet/pods/bdaf67ad-cce8-4dfe-9f82-48df7fcaed88/volumes"
Sep 16 21:46:06.346643 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:06.346608    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"]
Sep 16 21:46:06.347022 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:06.346904    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m" podUID="7f966d85-1c38-4502-bc7b-d36d78985f58" containerName="istio-proxy" containerID="cri-o://3515b914829d4880738c1143f53310f003e0622b186475a88fcb9bab80753f96" gracePeriod=30
Sep 16 21:46:07.628979 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.628941    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de41b-4xj99"]
Sep 16 21:46:07.629359 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.629347    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd82ee24-7331-4ced-9aaa-80b6a3739348" containerName="main"
Sep 16 21:46:07.629407 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.629362    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd82ee24-7331-4ced-9aaa-80b6a3739348" containerName="main"
Sep 16 21:46:07.629407 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.629381    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2429b0cb-93a4-4a8c-99a2-15aa7f127ada" containerName="main"
Sep 16 21:46:07.629407 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.629386    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2429b0cb-93a4-4a8c-99a2-15aa7f127ada" containerName="main"
Sep 16 21:46:07.629407 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.629395    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdaf67ad-cce8-4dfe-9f82-48df7fcaed88" containerName="main"
Sep 16 21:46:07.629407 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.629401    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdaf67ad-cce8-4dfe-9f82-48df7fcaed88" containerName="main"
Sep 16 21:46:07.629572 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.629410    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7" containerName="revoke-keys"
Sep 16 21:46:07.629572 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.629416    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7" containerName="revoke-keys"
Sep 16 21:46:07.629572 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.629486    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2429b0cb-93a4-4a8c-99a2-15aa7f127ada" containerName="main"
Sep 16 21:46:07.629572 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.629496    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdaf67ad-cce8-4dfe-9f82-48df7fcaed88" containerName="main"
Sep 16 21:46:07.629572 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.629503    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd82ee24-7331-4ced-9aaa-80b6a3739348" containerName="main"
Sep 16 21:46:07.629572 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.629510    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a8ea861-aa97-4b5c-88de-cc29d1f7b5f7" containerName="revoke-keys"
Sep 16 21:46:07.634096 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.634073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de41b-4xj99"
Sep 16 21:46:07.637200 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.637170    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:46:07.637340 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.637177    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:46:07.640561 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.640534    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de41b-4xj99"]
Sep 16 21:46:07.717273 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.717231    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89669f3-cb29-4a89-bc5a-b11b9a3aba86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0de41b-4xj99\" (UID: \"c89669f3-cb29-4a89-bc5a-b11b9a3aba86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de41b-4xj99"
Sep 16 21:46:07.817940 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.817906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89669f3-cb29-4a89-bc5a-b11b9a3aba86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0de41b-4xj99\" (UID: \"c89669f3-cb29-4a89-bc5a-b11b9a3aba86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de41b-4xj99"
Sep 16 21:46:07.818536 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.818516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89669f3-cb29-4a89-bc5a-b11b9a3aba86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0de41b-4xj99\" (UID: \"c89669f3-cb29-4a89-bc5a-b11b9a3aba86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de41b-4xj99"
Sep 16 21:46:07.947063 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:07.947031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de41b-4xj99"
Sep 16 21:46:08.217925 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:08.217902    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de41b-4xj99"]
Sep 16 21:46:08.220334 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:46:08.220299    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89669f3_cb29_4a89_bc5a_b11b9a3aba86.slice/crio-1a5af571aed5089574bb9241df63cfd46475cc630fef5ee8ac2c6dc4e3ffe0fc WatchSource:0}: Error finding container 1a5af571aed5089574bb9241df63cfd46475cc630fef5ee8ac2c6dc4e3ffe0fc: Status 404 returned error can't find the container with id 1a5af571aed5089574bb9241df63cfd46475cc630fef5ee8ac2c6dc4e3ffe0fc
Sep 16 21:46:09.071278 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:09.071242    2563 generic.go:356] "Generic (PLEG): container finished" podID="c89669f3-cb29-4a89-bc5a-b11b9a3aba86" containerID="a3c102ab31390a1b58aa6302ea9dbf5eb059ec90ea6751cc1e2f2f51da9193ae" exitCode=0
Sep 16 21:46:09.071676 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:09.071326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de41b-4xj99" event={"ID":"c89669f3-cb29-4a89-bc5a-b11b9a3aba86","Type":"ContainerDied","Data":"a3c102ab31390a1b58aa6302ea9dbf5eb059ec90ea6751cc1e2f2f51da9193ae"}
Sep 16 21:46:09.071676 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:09.071383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de41b-4xj99" event={"ID":"c89669f3-cb29-4a89-bc5a-b11b9a3aba86","Type":"ContainerStarted","Data":"1a5af571aed5089574bb9241df63cfd46475cc630fef5ee8ac2c6dc4e3ffe0fc"}
Sep 16 21:46:10.214188 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:10.214159    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de41b-4xj99"
Sep 16 21:46:10.343782 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:10.343697    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89669f3-cb29-4a89-bc5a-b11b9a3aba86-maas-api-service-ca\") pod \"c89669f3-cb29-4a89-bc5a-b11b9a3aba86\" (UID: \"c89669f3-cb29-4a89-bc5a-b11b9a3aba86\") "
Sep 16 21:46:10.344053 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:10.344031    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89669f3-cb29-4a89-bc5a-b11b9a3aba86-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c89669f3-cb29-4a89-bc5a-b11b9a3aba86" (UID: "c89669f3-cb29-4a89-bc5a-b11b9a3aba86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:46:10.445568 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:10.445534    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89669f3-cb29-4a89-bc5a-b11b9a3aba86-maas-api-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:11.083828 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.083798    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de41b-4xj99"
Sep 16 21:46:11.084003 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.083833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de41b-4xj99" event={"ID":"c89669f3-cb29-4a89-bc5a-b11b9a3aba86","Type":"ContainerDied","Data":"1a5af571aed5089574bb9241df63cfd46475cc630fef5ee8ac2c6dc4e3ffe0fc"}
Sep 16 21:46:11.084003 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.083886    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a5af571aed5089574bb9241df63cfd46475cc630fef5ee8ac2c6dc4e3ffe0fc"
Sep 16 21:46:11.608669 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.608636    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:46:11.656355 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.656267    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-token\") pod \"7f966d85-1c38-4502-bc7b-d36d78985f58\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") "
Sep 16 21:46:11.656514 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.656380    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f966d85-1c38-4502-bc7b-d36d78985f58-istiod-ca-cert\") pod \"7f966d85-1c38-4502-bc7b-d36d78985f58\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") "
Sep 16 21:46:11.656514 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.656416    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-workload-socket\") pod \"7f966d85-1c38-4502-bc7b-d36d78985f58\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") "
Sep 16 21:46:11.656514 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.656444    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-credential-socket\") pod \"7f966d85-1c38-4502-bc7b-d36d78985f58\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") "
Sep 16 21:46:11.656718 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.656533    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-podinfo\") pod \"7f966d85-1c38-4502-bc7b-d36d78985f58\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") "
Sep 16 21:46:11.656718 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.656579    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-workload-certs\") pod \"7f966d85-1c38-4502-bc7b-d36d78985f58\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") "
Sep 16 21:46:11.656718 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.656619    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-data\") pod \"7f966d85-1c38-4502-bc7b-d36d78985f58\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") "
Sep 16 21:46:11.656718 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.656705    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vntds\" (UniqueName: \"kubernetes.io/projected/7f966d85-1c38-4502-bc7b-d36d78985f58-kube-api-access-vntds\") pod \"7f966d85-1c38-4502-bc7b-d36d78985f58\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") "
Sep 16 21:46:11.656924 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.656744    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-envoy\") pod \"7f966d85-1c38-4502-bc7b-d36d78985f58\" (UID: \"7f966d85-1c38-4502-bc7b-d36d78985f58\") "
Sep 16 21:46:11.656924 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.656745    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f966d85-1c38-4502-bc7b-d36d78985f58" (UID: "7f966d85-1c38-4502-bc7b-d36d78985f58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:46:11.657028 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.656929    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f966d85-1c38-4502-bc7b-d36d78985f58" (UID: "7f966d85-1c38-4502-bc7b-d36d78985f58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:46:11.657028 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.656987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f966d85-1c38-4502-bc7b-d36d78985f58-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f966d85-1c38-4502-bc7b-d36d78985f58" (UID: "7f966d85-1c38-4502-bc7b-d36d78985f58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:46:11.657235 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.657199    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f966d85-1c38-4502-bc7b-d36d78985f58" (UID: "7f966d85-1c38-4502-bc7b-d36d78985f58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:46:11.657346 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.657329    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:11.657417 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.657350    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:11.657417 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.657365    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f966d85-1c38-4502-bc7b-d36d78985f58-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:11.657417 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.657379    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:11.657704 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.657531    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f966d85-1c38-4502-bc7b-d36d78985f58" (UID: "7f966d85-1c38-4502-bc7b-d36d78985f58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:46:11.658916 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.658896    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f966d85-1c38-4502-bc7b-d36d78985f58" (UID: "7f966d85-1c38-4502-bc7b-d36d78985f58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:46:11.659020 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.658986    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f966d85-1c38-4502-bc7b-d36d78985f58-kube-api-access-vntds" (OuterVolumeSpecName: "kube-api-access-vntds") pod "7f966d85-1c38-4502-bc7b-d36d78985f58" (UID: "7f966d85-1c38-4502-bc7b-d36d78985f58"). InnerVolumeSpecName "kube-api-access-vntds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:46:11.659337 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.659291    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f966d85-1c38-4502-bc7b-d36d78985f58" (UID: "7f966d85-1c38-4502-bc7b-d36d78985f58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:46:11.659416 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.659365    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f966d85-1c38-4502-bc7b-d36d78985f58" (UID: "7f966d85-1c38-4502-bc7b-d36d78985f58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:46:11.758615 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.758580    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:11.758615 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.758611    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:11.758615 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.758620    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vntds\" (UniqueName: \"kubernetes.io/projected/7f966d85-1c38-4502-bc7b-d36d78985f58-kube-api-access-vntds\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:11.758878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.758632    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:11.758878 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:11.758645    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f966d85-1c38-4502-bc7b-d36d78985f58-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:12.091571 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:12.091535    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f966d85-1c38-4502-bc7b-d36d78985f58" containerID="3515b914829d4880738c1143f53310f003e0622b186475a88fcb9bab80753f96" exitCode=0
Sep 16 21:46:12.091794 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:12.091589    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"
Sep 16 21:46:12.091794 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:12.091613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m" event={"ID":"7f966d85-1c38-4502-bc7b-d36d78985f58","Type":"ContainerDied","Data":"3515b914829d4880738c1143f53310f003e0622b186475a88fcb9bab80753f96"}
Sep 16 21:46:12.091794 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:12.091709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m" event={"ID":"7f966d85-1c38-4502-bc7b-d36d78985f58","Type":"ContainerDied","Data":"762f3afb0534c3d52306f1387682696329a2cf6c9a7ba82f5f0aa20c237fce74"}
Sep 16 21:46:12.091794 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:12.091745    2563 scope.go:117] "RemoveContainer" containerID="3515b914829d4880738c1143f53310f003e0622b186475a88fcb9bab80753f96"
Sep 16 21:46:12.104086 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:12.103784    2563 scope.go:117] "RemoveContainer" containerID="5dbad252c3a97baa344fc3cf5187ed474fa2dd7cd1ba717e3cb57e85ef212422"
Sep 16 21:46:12.117385 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:12.115289    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"]
Sep 16 21:46:12.124450 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:12.124173    2563 scope.go:117] "RemoveContainer" containerID="3515b914829d4880738c1143f53310f003e0622b186475a88fcb9bab80753f96"
Sep 16 21:46:12.125062 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:46:12.124734    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3515b914829d4880738c1143f53310f003e0622b186475a88fcb9bab80753f96\": container with ID starting with 3515b914829d4880738c1143f53310f003e0622b186475a88fcb9bab80753f96 not found: ID does not exist" containerID="3515b914829d4880738c1143f53310f003e0622b186475a88fcb9bab80753f96"
Sep 16 21:46:12.125062 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:12.124785    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3515b914829d4880738c1143f53310f003e0622b186475a88fcb9bab80753f96"} err="failed to get container status \"3515b914829d4880738c1143f53310f003e0622b186475a88fcb9bab80753f96\": rpc error: code = NotFound desc = could not find container \"3515b914829d4880738c1143f53310f003e0622b186475a88fcb9bab80753f96\": container with ID starting with 3515b914829d4880738c1143f53310f003e0622b186475a88fcb9bab80753f96 not found: ID does not exist"
Sep 16 21:46:12.125062 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:12.124810    2563 scope.go:117] "RemoveContainer" containerID="5dbad252c3a97baa344fc3cf5187ed474fa2dd7cd1ba717e3cb57e85ef212422"
Sep 16 21:46:12.125417 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:46:12.125338    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dbad252c3a97baa344fc3cf5187ed474fa2dd7cd1ba717e3cb57e85ef212422\": container with ID starting with 5dbad252c3a97baa344fc3cf5187ed474fa2dd7cd1ba717e3cb57e85ef212422 not found: ID does not exist" containerID="5dbad252c3a97baa344fc3cf5187ed474fa2dd7cd1ba717e3cb57e85ef212422"
Sep 16 21:46:12.125417 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:12.125368    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dbad252c3a97baa344fc3cf5187ed474fa2dd7cd1ba717e3cb57e85ef212422"} err="failed to get container status \"5dbad252c3a97baa344fc3cf5187ed474fa2dd7cd1ba717e3cb57e85ef212422\": rpc error: code = NotFound desc = could not find container \"5dbad252c3a97baa344fc3cf5187ed474fa2dd7cd1ba717e3cb57e85ef212422\": container with ID starting with 5dbad252c3a97baa344fc3cf5187ed474fa2dd7cd1ba717e3cb57e85ef212422 not found: ID does not exist"
Sep 16 21:46:12.127164 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:12.127144    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-35f5fd-openshift-default-68dcb7ddb-7m69m"]
Sep 16 21:46:12.610427 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:12.610374    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f966d85-1c38-4502-bc7b-d36d78985f58" path="/var/lib/kubelet/pods/7f966d85-1c38-4502-bc7b-d36d78985f58/volumes"
Sep 16 21:46:20.854209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:20.854169    2563 scope.go:117] "RemoveContainer" containerID="4c762f004b45498832927e36fd21ea761e7b59da5f21bad2055f42eb503564e7"
Sep 16 21:46:23.157074 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:23.156980    2563 generic.go:356] "Generic (PLEG): container finished" podID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" containerID="a42847133c9dde080d49326f99275c0c7f37420fb8c1318c27b8e3a237cc0e7d" exitCode=0
Sep 16 21:46:23.157074 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:23.157053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8" event={"ID":"11ca57ee-cf8a-41a5-9769-cc43a25f20af","Type":"ContainerDied","Data":"a42847133c9dde080d49326f99275c0c7f37420fb8c1318c27b8e3a237cc0e7d"}
Sep 16 21:46:23.157511 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:23.157484    2563 scope.go:117] "RemoveContainer" containerID="a42847133c9dde080d49326f99275c0c7f37420fb8c1318c27b8e3a237cc0e7d"
Sep 16 21:46:23.158668 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:23.158630    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:46:23.158730 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:23.158712    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:46:23.158765 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:23.158742    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:46:23.509311 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:23.509272    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de41b-4xj99"]
Sep 16 21:46:23.512336 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:23.512310    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de41b-4xj99"]
Sep 16 21:46:24.165006 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:24.164965    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8" event={"ID":"11ca57ee-cf8a-41a5-9769-cc43a25f20af","Type":"ContainerStarted","Data":"dc0d50a7fac04751abd348ac3ef0858407b7bc9b1fbdb26682af8c6bdd0e6beb"}
Sep 16 21:46:24.608371 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:24.608333    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c89669f3-cb29-4a89-bc5a-b11b9a3aba86" path="/var/lib/kubelet/pods/c89669f3-cb29-4a89-bc5a-b11b9a3aba86/volumes"
Sep 16 21:46:24.807703 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:24.807633    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:46:24.808750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:24.808723    2563 patch_prober.go:28] interesting pod/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 16 21:46:24.808877 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:24.808790    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8" podUID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 21:46:25.808504 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:25.808465    2563 patch_prober.go:28] interesting pod/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 16 21:46:25.809135 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:25.809101    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8" podUID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 21:46:26.808450 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:26.808418    2563 patch_prober.go:28] interesting pod/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 16 21:46:26.808626 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:26.808485    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8" podUID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 21:46:27.808295 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:27.808256    2563 patch_prober.go:28] interesting pod/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 16 21:46:27.808491 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:27.808330    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8" podUID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 21:46:28.808015 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:28.807979    2563 patch_prober.go:28] interesting pod/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 16 21:46:28.808404 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:28.808047    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8" podUID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 21:46:28.920542 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:28.920511    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:46:29.810749 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:29.810706    2563 patch_prober.go:28] interesting pod/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 16 21:46:29.811374 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:29.811336    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8" podUID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 21:46:30.009918 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.009879    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"]
Sep 16 21:46:30.010157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.010130    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" podUID="84974a56-ec81-435f-a8af-207b51fadedb" containerName="istio-proxy" containerID="cri-o://e559f1c9b8b4f80076093976c113da97dc8e62ae0671df84f883898421fa2fdb" gracePeriod=30
Sep 16 21:46:30.367369 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.367333    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0102fa-jvksc"]
Sep 16 21:46:30.367814 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.367794    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f966d85-1c38-4502-bc7b-d36d78985f58" containerName="istio-proxy"
Sep 16 21:46:30.367814 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.367815    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f966d85-1c38-4502-bc7b-d36d78985f58" containerName="istio-proxy"
Sep 16 21:46:30.367969 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.367862    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c89669f3-cb29-4a89-bc5a-b11b9a3aba86" containerName="revoke-keys"
Sep 16 21:46:30.367969 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.367870    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c89669f3-cb29-4a89-bc5a-b11b9a3aba86" containerName="revoke-keys"
Sep 16 21:46:30.368085 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.367975    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f966d85-1c38-4502-bc7b-d36d78985f58" containerName="istio-proxy"
Sep 16 21:46:30.368085 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.367989    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c89669f3-cb29-4a89-bc5a-b11b9a3aba86" containerName="revoke-keys"
Sep 16 21:46:30.368085 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.368001    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f966d85-1c38-4502-bc7b-d36d78985f58" containerName="istio-proxy"
Sep 16 21:46:30.371337 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.371315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0102fa-jvksc"
Sep 16 21:46:30.374839 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.374821    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:46:30.374946 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.374838    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:46:30.381266 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.381241    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0102fa-jvksc"]
Sep 16 21:46:30.464339 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.464293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae9659c0-42fb-4f2c-847b-50240ab5361a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0102fa-jvksc\" (UID: \"ae9659c0-42fb-4f2c-847b-50240ab5361a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0102fa-jvksc"
Sep 16 21:46:30.565980 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.565938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae9659c0-42fb-4f2c-847b-50240ab5361a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0102fa-jvksc\" (UID: \"ae9659c0-42fb-4f2c-847b-50240ab5361a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0102fa-jvksc"
Sep 16 21:46:30.566561 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.566540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae9659c0-42fb-4f2c-847b-50240ab5361a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0102fa-jvksc\" (UID: \"ae9659c0-42fb-4f2c-847b-50240ab5361a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0102fa-jvksc"
Sep 16 21:46:30.682418 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.682380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0102fa-jvksc"
Sep 16 21:46:30.812815 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.812783    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:46:30.813856 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.813834    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:46:30.961899 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:30.961876    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0102fa-jvksc"]
Sep 16 21:46:30.967667 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:46:30.967618    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9659c0_42fb_4f2c_847b_50240ab5361a.slice/crio-e9f0038c80dfa0f4958d10cf4526d34f7b7c6651e85f4e859779de2b51a41e0f WatchSource:0}: Error finding container e9f0038c80dfa0f4958d10cf4526d34f7b7c6651e85f4e859779de2b51a41e0f: Status 404 returned error can't find the container with id e9f0038c80dfa0f4958d10cf4526d34f7b7c6651e85f4e859779de2b51a41e0f
Sep 16 21:46:31.217773 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:31.217685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0102fa-jvksc" event={"ID":"ae9659c0-42fb-4f2c-847b-50240ab5361a","Type":"ContainerStarted","Data":"e9f0038c80dfa0f4958d10cf4526d34f7b7c6651e85f4e859779de2b51a41e0f"}
Sep 16 21:46:32.224309 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:32.224268    2563 generic.go:356] "Generic (PLEG): container finished" podID="ae9659c0-42fb-4f2c-847b-50240ab5361a" containerID="9ca6975a39d0e7b1071022a83647badb6e8310efa57e7c35b74e9a459d494246" exitCode=0
Sep 16 21:46:32.224798 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:32.224321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0102fa-jvksc" event={"ID":"ae9659c0-42fb-4f2c-847b-50240ab5361a","Type":"ContainerDied","Data":"9ca6975a39d0e7b1071022a83647badb6e8310efa57e7c35b74e9a459d494246"}
Sep 16 21:46:33.366124 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:33.366100    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0102fa-jvksc"
Sep 16 21:46:33.395385 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:33.395356    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae9659c0-42fb-4f2c-847b-50240ab5361a-maas-api-service-ca\") pod \"ae9659c0-42fb-4f2c-847b-50240ab5361a\" (UID: \"ae9659c0-42fb-4f2c-847b-50240ab5361a\") "
Sep 16 21:46:33.395731 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:33.395705    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae9659c0-42fb-4f2c-847b-50240ab5361a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae9659c0-42fb-4f2c-847b-50240ab5361a" (UID: "ae9659c0-42fb-4f2c-847b-50240ab5361a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:46:33.395846 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:33.395827    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae9659c0-42fb-4f2c-847b-50240ab5361a-maas-api-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:34.236405 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:34.236379    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0102fa-jvksc"
Sep 16 21:46:34.236590 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:34.236377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0102fa-jvksc" event={"ID":"ae9659c0-42fb-4f2c-847b-50240ab5361a","Type":"ContainerDied","Data":"e9f0038c80dfa0f4958d10cf4526d34f7b7c6651e85f4e859779de2b51a41e0f"}
Sep 16 21:46:34.236590 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:34.236499    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9f0038c80dfa0f4958d10cf4526d34f7b7c6651e85f4e859779de2b51a41e0f"
Sep 16 21:46:34.857704 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:34.857648    2563 patch_prober.go:28] interesting pod/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 21:46:34.858082 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:34.857721    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" podUID="84974a56-ec81-435f-a8af-207b51fadedb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 21:46:35.245253 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.245219    2563 generic.go:356] "Generic (PLEG): container finished" podID="84974a56-ec81-435f-a8af-207b51fadedb" containerID="e559f1c9b8b4f80076093976c113da97dc8e62ae0671df84f883898421fa2fdb" exitCode=0
Sep 16 21:46:35.245409 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.245285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" event={"ID":"84974a56-ec81-435f-a8af-207b51fadedb","Type":"ContainerDied","Data":"e559f1c9b8b4f80076093976c113da97dc8e62ae0671df84f883898421fa2fdb"}
Sep 16 21:46:35.245409 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.245346    2563 scope.go:117] "RemoveContainer" containerID="f2f22226b91362faeee63e7ae083fc763296d3f8eaf5607b5e40fb6be7567a5f"
Sep 16 21:46:35.268310 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:46:35.268287    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f22226b91362faeee63e7ae083fc763296d3f8eaf5607b5e40fb6be7567a5f\": container with ID starting with f2f22226b91362faeee63e7ae083fc763296d3f8eaf5607b5e40fb6be7567a5f not found: ID does not exist" containerID="f2f22226b91362faeee63e7ae083fc763296d3f8eaf5607b5e40fb6be7567a5f"
Sep 16 21:46:35.271468 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.271449    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:46:35.315741 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.315711    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4s6z\" (UniqueName: \"kubernetes.io/projected/84974a56-ec81-435f-a8af-207b51fadedb-kube-api-access-p4s6z\") pod \"84974a56-ec81-435f-a8af-207b51fadedb\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") "
Sep 16 21:46:35.315924 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.315752    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-istio-data\") pod \"84974a56-ec81-435f-a8af-207b51fadedb\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") "
Sep 16 21:46:35.315924 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.315828    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-istio-envoy\") pod \"84974a56-ec81-435f-a8af-207b51fadedb\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") "
Sep 16 21:46:35.315924 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.315862    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-credential-socket\") pod \"84974a56-ec81-435f-a8af-207b51fadedb\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") "
Sep 16 21:46:35.315924 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.315906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84974a56-ec81-435f-a8af-207b51fadedb-istio-token\") pod \"84974a56-ec81-435f-a8af-207b51fadedb\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") "
Sep 16 21:46:35.316147 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.315949    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84974a56-ec81-435f-a8af-207b51fadedb-istio-podinfo\") pod \"84974a56-ec81-435f-a8af-207b51fadedb\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") "
Sep 16 21:46:35.316147 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.316029    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-workload-socket\") pod \"84974a56-ec81-435f-a8af-207b51fadedb\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") "
Sep 16 21:46:35.316147 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.316052    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-istio-data" (OuterVolumeSpecName: "istio-data") pod "84974a56-ec81-435f-a8af-207b51fadedb" (UID: "84974a56-ec81-435f-a8af-207b51fadedb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:46:35.316147 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.316087    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-workload-certs\") pod \"84974a56-ec81-435f-a8af-207b51fadedb\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") "
Sep 16 21:46:35.316147 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.316129    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84974a56-ec81-435f-a8af-207b51fadedb-istiod-ca-cert\") pod \"84974a56-ec81-435f-a8af-207b51fadedb\" (UID: \"84974a56-ec81-435f-a8af-207b51fadedb\") "
Sep 16 21:46:35.316377 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.316226    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "84974a56-ec81-435f-a8af-207b51fadedb" (UID: "84974a56-ec81-435f-a8af-207b51fadedb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:46:35.316472 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.316424    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "84974a56-ec81-435f-a8af-207b51fadedb" (UID: "84974a56-ec81-435f-a8af-207b51fadedb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:46:35.316683 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.316561    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:35.316683 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.316585    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:35.316683 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.316628    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:35.316951 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.316787    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84974a56-ec81-435f-a8af-207b51fadedb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "84974a56-ec81-435f-a8af-207b51fadedb" (UID: "84974a56-ec81-435f-a8af-207b51fadedb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:46:35.317243 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.317220    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "84974a56-ec81-435f-a8af-207b51fadedb" (UID: "84974a56-ec81-435f-a8af-207b51fadedb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:46:35.318483 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.318451    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84974a56-ec81-435f-a8af-207b51fadedb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "84974a56-ec81-435f-a8af-207b51fadedb" (UID: "84974a56-ec81-435f-a8af-207b51fadedb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:46:35.318640 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.318611    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84974a56-ec81-435f-a8af-207b51fadedb-kube-api-access-p4s6z" (OuterVolumeSpecName: "kube-api-access-p4s6z") pod "84974a56-ec81-435f-a8af-207b51fadedb" (UID: "84974a56-ec81-435f-a8af-207b51fadedb"). InnerVolumeSpecName "kube-api-access-p4s6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:46:35.318768 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.318626    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84974a56-ec81-435f-a8af-207b51fadedb-istio-token" (OuterVolumeSpecName: "istio-token") pod "84974a56-ec81-435f-a8af-207b51fadedb" (UID: "84974a56-ec81-435f-a8af-207b51fadedb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:46:35.319304 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.319278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "84974a56-ec81-435f-a8af-207b51fadedb" (UID: "84974a56-ec81-435f-a8af-207b51fadedb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:46:35.417282 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.417173    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4s6z\" (UniqueName: \"kubernetes.io/projected/84974a56-ec81-435f-a8af-207b51fadedb-kube-api-access-p4s6z\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:35.417282 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.417211    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:35.417282 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.417224    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84974a56-ec81-435f-a8af-207b51fadedb-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:35.417282 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.417235    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84974a56-ec81-435f-a8af-207b51fadedb-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:35.417282 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.417246    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84974a56-ec81-435f-a8af-207b51fadedb-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:35.417282 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:35.417258    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84974a56-ec81-435f-a8af-207b51fadedb-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:36.252622 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:36.252584    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt" event={"ID":"84974a56-ec81-435f-a8af-207b51fadedb","Type":"ContainerDied","Data":"f31c7118f7cbc594b8caff558ca27766c9ea3ee9f41e8eacb6eda6a8524f3542"}
Sep 16 21:46:36.253082 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:36.252645    2563 scope.go:117] "RemoveContainer" containerID="e559f1c9b8b4f80076093976c113da97dc8e62ae0671df84f883898421fa2fdb"
Sep 16 21:46:36.253082 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:36.252599    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"
Sep 16 21:46:36.274590 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:36.274556    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"]
Sep 16 21:46:36.276665 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:36.276633    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0de41b-openshift-default-7f69dd45dd-hthqt"]
Sep 16 21:46:36.608193 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:36.608115    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84974a56-ec81-435f-a8af-207b51fadedb" path="/var/lib/kubelet/pods/84974a56-ec81-435f-a8af-207b51fadedb/volumes"
Sep 16 21:46:46.115327 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:46.115289    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0102fa-jvksc"]
Sep 16 21:46:46.117970 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:46.117944    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0102fa-jvksc"]
Sep 16 21:46:46.609371 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:46.609333    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae9659c0-42fb-4f2c-847b-50240ab5361a" path="/var/lib/kubelet/pods/ae9659c0-42fb-4f2c-847b-50240ab5361a/volumes"
Sep 16 21:46:53.206595 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.206556    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"]
Sep 16 21:46:53.207068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.206834    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8" podUID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" containerName="istio-proxy" containerID="cri-o://dc0d50a7fac04751abd348ac3ef0858407b7bc9b1fbdb26682af8c6bdd0e6beb" gracePeriod=30
Sep 16 21:46:53.545465 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.545381    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz"]
Sep 16 21:46:53.545816 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.545801    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f966d85-1c38-4502-bc7b-d36d78985f58" containerName="istio-proxy"
Sep 16 21:46:53.545877 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.545818    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f966d85-1c38-4502-bc7b-d36d78985f58" containerName="istio-proxy"
Sep 16 21:46:53.545877 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.545843    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84974a56-ec81-435f-a8af-207b51fadedb" containerName="istio-proxy"
Sep 16 21:46:53.545877 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.545849    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="84974a56-ec81-435f-a8af-207b51fadedb" containerName="istio-proxy"
Sep 16 21:46:53.545877 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.545859    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae9659c0-42fb-4f2c-847b-50240ab5361a" containerName="revoke-keys"
Sep 16 21:46:53.545877 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.545865    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae9659c0-42fb-4f2c-847b-50240ab5361a" containerName="revoke-keys"
Sep 16 21:46:53.545877 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.545872    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84974a56-ec81-435f-a8af-207b51fadedb" containerName="istio-proxy"
Sep 16 21:46:53.545877 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.545877    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="84974a56-ec81-435f-a8af-207b51fadedb" containerName="istio-proxy"
Sep 16 21:46:53.546089 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.545934    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="84974a56-ec81-435f-a8af-207b51fadedb" containerName="istio-proxy"
Sep 16 21:46:53.546089 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.545941    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae9659c0-42fb-4f2c-847b-50240ab5361a" containerName="revoke-keys"
Sep 16 21:46:53.546089 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.545948    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="84974a56-ec81-435f-a8af-207b51fadedb" containerName="istio-proxy"
Sep 16 21:46:53.548723 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.548707    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz"
Sep 16 21:46:53.551114 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.551088    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:46:53.551351 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.551332    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:46:53.558393 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.558364    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz"]
Sep 16 21:46:53.718960 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.718917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d94be47-ca97-4fef-b2fb-13568aba6b9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz\" (UID: \"4d94be47-ca97-4fef-b2fb-13568aba6b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz"
Sep 16 21:46:53.820218 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.820114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d94be47-ca97-4fef-b2fb-13568aba6b9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz\" (UID: \"4d94be47-ca97-4fef-b2fb-13568aba6b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz"
Sep 16 21:46:53.821038 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.821013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d94be47-ca97-4fef-b2fb-13568aba6b9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz\" (UID: \"4d94be47-ca97-4fef-b2fb-13568aba6b9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz"
Sep 16 21:46:53.862031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:53.861994    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz"
Sep 16 21:46:54.145477 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:54.145445    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz"]
Sep 16 21:46:54.354241 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:54.354207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz" event={"ID":"4d94be47-ca97-4fef-b2fb-13568aba6b9d","Type":"ContainerStarted","Data":"643f82651c1cf1bc7b0078aa427e897afa6ef565d614fed8c46b5aac462e707f"}
Sep 16 21:46:55.362079 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:55.362047    2563 generic.go:356] "Generic (PLEG): container finished" podID="4d94be47-ca97-4fef-b2fb-13568aba6b9d" containerID="627c48812a62f6ef7466f6a1253eea13af8c1c9f15ede0720dfbabf16cb483f3" exitCode=0
Sep 16 21:46:55.362533 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:55.362133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz" event={"ID":"4d94be47-ca97-4fef-b2fb-13568aba6b9d","Type":"ContainerDied","Data":"627c48812a62f6ef7466f6a1253eea13af8c1c9f15ede0720dfbabf16cb483f3"}
Sep 16 21:46:56.503220 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:56.503193    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz"
Sep 16 21:46:56.649521 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:56.649427    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d94be47-ca97-4fef-b2fb-13568aba6b9d-maas-api-service-ca\") pod \"4d94be47-ca97-4fef-b2fb-13568aba6b9d\" (UID: \"4d94be47-ca97-4fef-b2fb-13568aba6b9d\") "
Sep 16 21:46:56.649849 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:56.649823    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d94be47-ca97-4fef-b2fb-13568aba6b9d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4d94be47-ca97-4fef-b2fb-13568aba6b9d" (UID: "4d94be47-ca97-4fef-b2fb-13568aba6b9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:46:56.650002 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:56.649989    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d94be47-ca97-4fef-b2fb-13568aba6b9d-maas-api-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:57.373873 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:57.373833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz" event={"ID":"4d94be47-ca97-4fef-b2fb-13568aba6b9d","Type":"ContainerDied","Data":"643f82651c1cf1bc7b0078aa427e897afa6ef565d614fed8c46b5aac462e707f"}
Sep 16 21:46:57.374027 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:57.373890    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="643f82651c1cf1bc7b0078aa427e897afa6ef565d614fed8c46b5aac462e707f"
Sep 16 21:46:57.374027 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:57.373855    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz"
Sep 16 21:46:58.382896 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.382868    2563 generic.go:356] "Generic (PLEG): container finished" podID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" containerID="dc0d50a7fac04751abd348ac3ef0858407b7bc9b1fbdb26682af8c6bdd0e6beb" exitCode=0
Sep 16 21:46:58.383311 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.382970    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8" event={"ID":"11ca57ee-cf8a-41a5-9769-cc43a25f20af","Type":"ContainerDied","Data":"dc0d50a7fac04751abd348ac3ef0858407b7bc9b1fbdb26682af8c6bdd0e6beb"}
Sep 16 21:46:58.383311 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.383038    2563 scope.go:117] "RemoveContainer" containerID="a42847133c9dde080d49326f99275c0c7f37420fb8c1318c27b8e3a237cc0e7d"
Sep 16 21:46:58.475352 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.475288    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:46:58.671749 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.671709    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-data\") pod \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") "
Sep 16 21:46:58.671749 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.671748    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-credential-socket\") pod \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") "
Sep 16 21:46:58.671991 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.671789    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-envoy\") pod \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") "
Sep 16 21:46:58.671991 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.671817    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-podinfo\") pod \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") "
Sep 16 21:46:58.671991 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.671843    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istiod-ca-cert\") pod \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") "
Sep 16 21:46:58.671991 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.671886    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-workload-certs\") pod \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") "
Sep 16 21:46:58.671991 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.671952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djcmh\" (UniqueName: \"kubernetes.io/projected/11ca57ee-cf8a-41a5-9769-cc43a25f20af-kube-api-access-djcmh\") pod \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") "
Sep 16 21:46:58.672245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.672043    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-workload-socket\") pod \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") "
Sep 16 21:46:58.672245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.672089    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-token\") pod \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\" (UID: \"11ca57ee-cf8a-41a5-9769-cc43a25f20af\") "
Sep 16 21:46:58.672245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.672040    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-data" (OuterVolumeSpecName: "istio-data") pod "11ca57ee-cf8a-41a5-9769-cc43a25f20af" (UID: "11ca57ee-cf8a-41a5-9769-cc43a25f20af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:46:58.672245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.672095    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11ca57ee-cf8a-41a5-9769-cc43a25f20af" (UID: "11ca57ee-cf8a-41a5-9769-cc43a25f20af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:46:58.672455 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.672265    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11ca57ee-cf8a-41a5-9769-cc43a25f20af" (UID: "11ca57ee-cf8a-41a5-9769-cc43a25f20af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:46:58.672455 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.672302    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11ca57ee-cf8a-41a5-9769-cc43a25f20af" (UID: "11ca57ee-cf8a-41a5-9769-cc43a25f20af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:46:58.672632 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.672578    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11ca57ee-cf8a-41a5-9769-cc43a25f20af" (UID: "11ca57ee-cf8a-41a5-9769-cc43a25f20af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:46:58.672886 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.672868    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:58.672974 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.672893    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:58.672974 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.672907    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:58.672974 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.672920    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:58.672974 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.672932    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:58.674293 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.674271    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11ca57ee-cf8a-41a5-9769-cc43a25f20af" (UID: "11ca57ee-cf8a-41a5-9769-cc43a25f20af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:46:58.674366 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.674290    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11ca57ee-cf8a-41a5-9769-cc43a25f20af" (UID: "11ca57ee-cf8a-41a5-9769-cc43a25f20af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:46:58.674544 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.674524    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ca57ee-cf8a-41a5-9769-cc43a25f20af-kube-api-access-djcmh" (OuterVolumeSpecName: "kube-api-access-djcmh") pod "11ca57ee-cf8a-41a5-9769-cc43a25f20af" (UID: "11ca57ee-cf8a-41a5-9769-cc43a25f20af"). InnerVolumeSpecName "kube-api-access-djcmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:46:58.674606 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.674552    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-token" (OuterVolumeSpecName: "istio-token") pod "11ca57ee-cf8a-41a5-9769-cc43a25f20af" (UID: "11ca57ee-cf8a-41a5-9769-cc43a25f20af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:46:58.773639 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.773548    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:58.773639 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.773584    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djcmh\" (UniqueName: \"kubernetes.io/projected/11ca57ee-cf8a-41a5-9769-cc43a25f20af-kube-api-access-djcmh\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:58.773639 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.773598    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:58.773639 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:58.773612    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ca57ee-cf8a-41a5-9769-cc43a25f20af-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:46:59.389978 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:59.389949    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"
Sep 16 21:46:59.390393 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:59.389993    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8" event={"ID":"11ca57ee-cf8a-41a5-9769-cc43a25f20af","Type":"ContainerDied","Data":"c985bde1566eaa5dc1518f67fd443ef87c0c2de67aa00fb8eee1a99cf36e7747"}
Sep 16 21:46:59.390393 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:59.390056    2563 scope.go:117] "RemoveContainer" containerID="dc0d50a7fac04751abd348ac3ef0858407b7bc9b1fbdb26682af8c6bdd0e6beb"
Sep 16 21:46:59.411502 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:59.411474    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"]
Sep 16 21:46:59.414839 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:46:59.414816    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0102fa-openshift-default-6b8fd5fdb-kzzj8"]
Sep 16 21:47:00.608706 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:00.608674    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" path="/var/lib/kubelet/pods/11ca57ee-cf8a-41a5-9769-cc43a25f20af/volumes"
Sep 16 21:47:04.328482 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:04.328402    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz"]
Sep 16 21:47:04.330999 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:04.330975    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-296002-d7pnz"]
Sep 16 21:47:04.608941 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:04.608853    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d94be47-ca97-4fef-b2fb-13568aba6b9d" path="/var/lib/kubelet/pods/4d94be47-ca97-4fef-b2fb-13568aba6b9d/volumes"
Sep 16 21:47:10.882317 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:10.882271    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"]
Sep 16 21:47:10.882800 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:10.882557    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp" podUID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" containerName="istio-proxy" containerID="cri-o://60d64d5eecfc93d3be8faf49e75ac30643c84c1a033af46520d09fd4de5e750f" gracePeriod=30
Sep 16 21:47:11.267921 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.267863    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz"]
Sep 16 21:47:11.269003 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.268969    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" containerName="istio-proxy"
Sep 16 21:47:11.269145 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.269134    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" containerName="istio-proxy"
Sep 16 21:47:11.269237 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.269227    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d94be47-ca97-4fef-b2fb-13568aba6b9d" containerName="revoke-keys"
Sep 16 21:47:11.269312 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.269303    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d94be47-ca97-4fef-b2fb-13568aba6b9d" containerName="revoke-keys"
Sep 16 21:47:11.269600 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.269585    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" containerName="istio-proxy"
Sep 16 21:47:11.269729 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.269718    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d94be47-ca97-4fef-b2fb-13568aba6b9d" containerName="revoke-keys"
Sep 16 21:47:11.275123 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.275097    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz"
Sep 16 21:47:11.283132 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.283107    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:47:11.283378 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.283141    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:47:11.284008 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.283984    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz"]
Sep 16 21:47:11.401039 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.401001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d55c170-b530-455c-b557-128b57c4517d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz\" (UID: \"1d55c170-b530-455c-b557-128b57c4517d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz"
Sep 16 21:47:11.502103 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.502066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d55c170-b530-455c-b557-128b57c4517d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz\" (UID: \"1d55c170-b530-455c-b557-128b57c4517d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz"
Sep 16 21:47:11.502708 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.502681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d55c170-b530-455c-b557-128b57c4517d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz\" (UID: \"1d55c170-b530-455c-b557-128b57c4517d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz"
Sep 16 21:47:11.586866 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.586774    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz"
Sep 16 21:47:11.866183 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:11.866026    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz"]
Sep 16 21:47:12.465912 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:12.465821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz" event={"ID":"1d55c170-b530-455c-b557-128b57c4517d","Type":"ContainerStarted","Data":"f7194236b378809fa5ae65e8f0fb60f9117c501734cd66b9c9f2c5f02fad087b"}
Sep 16 21:47:12.465912 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:12.465889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz" event={"ID":"1d55c170-b530-455c-b557-128b57c4517d","Type":"ContainerStarted","Data":"46426bed91dff1dbd1607a10b0657d2c9c767bf5cbb8bba394fee7a3d9edccab"}
Sep 16 21:47:12.482129 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:12.482080    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz" podStartSLOduration=1.482064834 podStartE2EDuration="1.482064834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:47:12.479892862 +0000 UTC m=+1492.380102865" watchObservedRunningTime="2026-09-16 21:47:12.482064834 +0000 UTC m=+1492.382274838"
Sep 16 21:47:13.472338 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:13.472298    2563 generic.go:356] "Generic (PLEG): container finished" podID="1d55c170-b530-455c-b557-128b57c4517d" containerID="f7194236b378809fa5ae65e8f0fb60f9117c501734cd66b9c9f2c5f02fad087b" exitCode=0
Sep 16 21:47:13.472815 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:13.472385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz" event={"ID":"1d55c170-b530-455c-b557-128b57c4517d","Type":"ContainerDied","Data":"f7194236b378809fa5ae65e8f0fb60f9117c501734cd66b9c9f2c5f02fad087b"}
Sep 16 21:47:14.608338 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:14.608308    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz"
Sep 16 21:47:14.737193 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:14.737102    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d55c170-b530-455c-b557-128b57c4517d-maas-api-service-ca\") pod \"1d55c170-b530-455c-b557-128b57c4517d\" (UID: \"1d55c170-b530-455c-b557-128b57c4517d\") "
Sep 16 21:47:14.737458 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:14.737434    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d55c170-b530-455c-b557-128b57c4517d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d55c170-b530-455c-b557-128b57c4517d" (UID: "1d55c170-b530-455c-b557-128b57c4517d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:47:14.838140 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:14.838105    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d55c170-b530-455c-b557-128b57c4517d-maas-api-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:15.484954 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.484867    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz"
Sep 16 21:47:15.484954 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.484904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz" event={"ID":"1d55c170-b530-455c-b557-128b57c4517d","Type":"ContainerDied","Data":"46426bed91dff1dbd1607a10b0657d2c9c767bf5cbb8bba394fee7a3d9edccab"}
Sep 16 21:47:15.484954 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.484956    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46426bed91dff1dbd1607a10b0657d2c9c767bf5cbb8bba394fee7a3d9edccab"
Sep 16 21:47:15.671836 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.671794    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4"]
Sep 16 21:47:15.672238 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.672175    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d55c170-b530-455c-b557-128b57c4517d" containerName="revoke-keys"
Sep 16 21:47:15.672238 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.672186    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d55c170-b530-455c-b557-128b57c4517d" containerName="revoke-keys"
Sep 16 21:47:15.672238 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.672201    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" containerName="istio-proxy"
Sep 16 21:47:15.672238 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.672210    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" containerName="istio-proxy"
Sep 16 21:47:15.672397 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.672269    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d55c170-b530-455c-b557-128b57c4517d" containerName="revoke-keys"
Sep 16 21:47:15.672397 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.672276    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="11ca57ee-cf8a-41a5-9769-cc43a25f20af" containerName="istio-proxy"
Sep 16 21:47:15.675355 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.675334    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4"
Sep 16 21:47:15.677580 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.677559    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:47:15.677747 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.677729    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:47:15.682500 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.682478    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4"]
Sep 16 21:47:15.747813 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.747715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4\" (UID: \"f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4"
Sep 16 21:47:15.849148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.849109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4\" (UID: \"f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4"
Sep 16 21:47:15.849810 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.849791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4\" (UID: \"f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4"
Sep 16 21:47:15.988318 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:15.988289    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4"
Sep 16 21:47:16.158522 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.158490    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:47:16.255161 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.255079    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gqgr\" (UniqueName: \"kubernetes.io/projected/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-kube-api-access-7gqgr\") pod \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") "
Sep 16 21:47:16.255161 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.255133    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-envoy\") pod \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") "
Sep 16 21:47:16.255401 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.255274    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-workload-certs\") pod \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") "
Sep 16 21:47:16.255401 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.255331    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-podinfo\") pod \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") "
Sep 16 21:47:16.255401 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.255363    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-workload-socket\") pod \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") "
Sep 16 21:47:16.255401 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.255387    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-credential-socket\") pod \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") "
Sep 16 21:47:16.255613 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.255408    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d7831d5-d49f-41fe-9e06-c3f8c1027b11" (UID: "3d7831d5-d49f-41fe-9e06-c3f8c1027b11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:16.255613 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.255511    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-token\") pod \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") "
Sep 16 21:47:16.255613 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.255543    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istiod-ca-cert\") pod \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") "
Sep 16 21:47:16.255810 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.255625    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d7831d5-d49f-41fe-9e06-c3f8c1027b11" (UID: "3d7831d5-d49f-41fe-9e06-c3f8c1027b11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:16.255810 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.255639    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d7831d5-d49f-41fe-9e06-c3f8c1027b11" (UID: "3d7831d5-d49f-41fe-9e06-c3f8c1027b11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:16.255810 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.255702    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-data\") pod \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\" (UID: \"3d7831d5-d49f-41fe-9e06-c3f8c1027b11\") "
Sep 16 21:47:16.255972 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.255941    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d7831d5-d49f-41fe-9e06-c3f8c1027b11" (UID: "3d7831d5-d49f-41fe-9e06-c3f8c1027b11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:16.256031 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.255972    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d7831d5-d49f-41fe-9e06-c3f8c1027b11" (UID: "3d7831d5-d49f-41fe-9e06-c3f8c1027b11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:47:16.256177 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.256155    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:16.256270 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.256180    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:16.256270 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.256197    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:16.256270 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.256211    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:16.256270 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.256226    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:16.257917 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.257630    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-kube-api-access-7gqgr" (OuterVolumeSpecName: "kube-api-access-7gqgr") pod "3d7831d5-d49f-41fe-9e06-c3f8c1027b11" (UID: "3d7831d5-d49f-41fe-9e06-c3f8c1027b11"). InnerVolumeSpecName "kube-api-access-7gqgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:16.257917 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.257875    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d7831d5-d49f-41fe-9e06-c3f8c1027b11" (UID: "3d7831d5-d49f-41fe-9e06-c3f8c1027b11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:47:16.258121 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.257999    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d7831d5-d49f-41fe-9e06-c3f8c1027b11" (UID: "3d7831d5-d49f-41fe-9e06-c3f8c1027b11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:16.258505 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.258478    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d7831d5-d49f-41fe-9e06-c3f8c1027b11" (UID: "3d7831d5-d49f-41fe-9e06-c3f8c1027b11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:16.261062 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.261043    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4"]
Sep 16 21:47:16.262849 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:47:16.262824    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ccbceb_b8a1_433e_aeb8_5aaa70b828e5.slice/crio-0bf4ce89338288c158ac7dd4ed3f52c3f38afca04536142ea2d4ee945e99acd6 WatchSource:0}: Error finding container 0bf4ce89338288c158ac7dd4ed3f52c3f38afca04536142ea2d4ee945e99acd6: Status 404 returned error can't find the container with id 0bf4ce89338288c158ac7dd4ed3f52c3f38afca04536142ea2d4ee945e99acd6
Sep 16 21:47:16.357420 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.357379    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gqgr\" (UniqueName: \"kubernetes.io/projected/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-kube-api-access-7gqgr\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:16.357420 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.357409    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:16.357420 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.357425    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:16.357651 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.357433    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7831d5-d49f-41fe-9e06-c3f8c1027b11-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:16.498491 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.498455    2563 generic.go:356] "Generic (PLEG): container finished" podID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" containerID="60d64d5eecfc93d3be8faf49e75ac30643c84c1a033af46520d09fd4de5e750f" exitCode=0
Sep 16 21:47:16.498683 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.498520    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"
Sep 16 21:47:16.498683 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.498538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp" event={"ID":"3d7831d5-d49f-41fe-9e06-c3f8c1027b11","Type":"ContainerDied","Data":"60d64d5eecfc93d3be8faf49e75ac30643c84c1a033af46520d09fd4de5e750f"}
Sep 16 21:47:16.498683 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.498633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp" event={"ID":"3d7831d5-d49f-41fe-9e06-c3f8c1027b11","Type":"ContainerDied","Data":"aef5fc155336486c8aae11650c5ade3936a555356008b68dff25c1689ef02405"}
Sep 16 21:47:16.498683 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.498682    2563 scope.go:117] "RemoveContainer" containerID="60d64d5eecfc93d3be8faf49e75ac30643c84c1a033af46520d09fd4de5e750f"
Sep 16 21:47:16.500110 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.499988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4" event={"ID":"f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5","Type":"ContainerStarted","Data":"0bf4ce89338288c158ac7dd4ed3f52c3f38afca04536142ea2d4ee945e99acd6"}
Sep 16 21:47:16.508803 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.508785    2563 scope.go:117] "RemoveContainer" containerID="3816449b88cf097d55a65f821cca63c3da8b07985f8c8b158d9e81acee028d8b"
Sep 16 21:47:16.517592 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.517569    2563 scope.go:117] "RemoveContainer" containerID="60d64d5eecfc93d3be8faf49e75ac30643c84c1a033af46520d09fd4de5e750f"
Sep 16 21:47:16.517914 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:47:16.517894    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d64d5eecfc93d3be8faf49e75ac30643c84c1a033af46520d09fd4de5e750f\": container with ID starting with 60d64d5eecfc93d3be8faf49e75ac30643c84c1a033af46520d09fd4de5e750f not found: ID does not exist" containerID="60d64d5eecfc93d3be8faf49e75ac30643c84c1a033af46520d09fd4de5e750f"
Sep 16 21:47:16.517989 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.517927    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d64d5eecfc93d3be8faf49e75ac30643c84c1a033af46520d09fd4de5e750f"} err="failed to get container status \"60d64d5eecfc93d3be8faf49e75ac30643c84c1a033af46520d09fd4de5e750f\": rpc error: code = NotFound desc = could not find container \"60d64d5eecfc93d3be8faf49e75ac30643c84c1a033af46520d09fd4de5e750f\": container with ID starting with 60d64d5eecfc93d3be8faf49e75ac30643c84c1a033af46520d09fd4de5e750f not found: ID does not exist"
Sep 16 21:47:16.517989 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.517955    2563 scope.go:117] "RemoveContainer" containerID="3816449b88cf097d55a65f821cca63c3da8b07985f8c8b158d9e81acee028d8b"
Sep 16 21:47:16.518255 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:47:16.518236    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3816449b88cf097d55a65f821cca63c3da8b07985f8c8b158d9e81acee028d8b\": container with ID starting with 3816449b88cf097d55a65f821cca63c3da8b07985f8c8b158d9e81acee028d8b not found: ID does not exist" containerID="3816449b88cf097d55a65f821cca63c3da8b07985f8c8b158d9e81acee028d8b"
Sep 16 21:47:16.518293 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.518261    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3816449b88cf097d55a65f821cca63c3da8b07985f8c8b158d9e81acee028d8b"} err="failed to get container status \"3816449b88cf097d55a65f821cca63c3da8b07985f8c8b158d9e81acee028d8b\": rpc error: code = NotFound desc = could not find container \"3816449b88cf097d55a65f821cca63c3da8b07985f8c8b158d9e81acee028d8b\": container with ID starting with 3816449b88cf097d55a65f821cca63c3da8b07985f8c8b158d9e81acee028d8b not found: ID does not exist"
Sep 16 21:47:16.522746 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.522719    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"]
Sep 16 21:47:16.529090 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.529066    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-296002-openshift-default-7674c75784-94twp"]
Sep 16 21:47:16.609302 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:16.609271    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" path="/var/lib/kubelet/pods/3d7831d5-d49f-41fe-9e06-c3f8c1027b11/volumes"
Sep 16 21:47:17.508229 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:17.508195    2563 generic.go:356] "Generic (PLEG): container finished" podID="f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5" containerID="63bf86fa942994d639253215f54fb1d762755774921fa13bb466313629d11ddb" exitCode=0
Sep 16 21:47:17.508735 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:17.508286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4" event={"ID":"f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5","Type":"ContainerDied","Data":"63bf86fa942994d639253215f54fb1d762755774921fa13bb466313629d11ddb"}
Sep 16 21:47:18.645982 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:18.645959    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4"
Sep 16 21:47:18.778366 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:18.778269    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5-maas-api-service-ca\") pod \"f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5\" (UID: \"f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5\") "
Sep 16 21:47:18.778747 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:18.778712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5" (UID: "f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:47:18.879545 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:18.879508    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5-maas-api-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:19.520068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:19.520035    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4"
Sep 16 21:47:19.520230 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:19.520071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4" event={"ID":"f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5","Type":"ContainerDied","Data":"0bf4ce89338288c158ac7dd4ed3f52c3f38afca04536142ea2d4ee945e99acd6"}
Sep 16 21:47:19.520230 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:19.520124    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bf4ce89338288c158ac7dd4ed3f52c3f38afca04536142ea2d4ee945e99acd6"
Sep 16 21:47:20.566219 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:20.566184    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"]
Sep 16 21:47:22.031324 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:22.031287    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz"]
Sep 16 21:47:22.037029 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:22.037002    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-365ea1-l77mz"]
Sep 16 21:47:22.608621 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:22.608576    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d55c170-b530-455c-b557-128b57c4517d" path="/var/lib/kubelet/pods/1d55c170-b530-455c-b557-128b57c4517d/volumes"
Sep 16 21:47:28.348245 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:28.348206    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"]
Sep 16 21:47:28.348714 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:28.348462    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl" podUID="6553a042-53fe-40f5-843c-4815273f5737" containerName="istio-proxy" containerID="cri-o://0427ecb74d2eb742841649bb4749b202407ca907a8eab342740f491ac815d1e0" gracePeriod=30
Sep 16 21:47:31.183029 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:31.182995    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e047d8-29826585-n8pjr"]
Sep 16 21:47:31.185750 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:31.185725    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-e047d8-29826585-n8pjr"]
Sep 16 21:47:31.477123 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:31.477034    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4"]
Sep 16 21:47:31.480013 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:31.479984    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e047d8-zwgf4"]
Sep 16 21:47:32.607513 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:32.607482    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25ae327a-e0f1-4e4a-8303-a4d549a1fc8a" path="/var/lib/kubelet/pods/25ae327a-e0f1-4e4a-8303-a4d549a1fc8a/volumes"
Sep 16 21:47:32.607921 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:32.607817    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5" path="/var/lib/kubelet/pods/f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5/volumes"
Sep 16 21:47:33.603537 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.603511    2563 generic.go:356] "Generic (PLEG): container finished" podID="6553a042-53fe-40f5-843c-4815273f5737" containerID="0427ecb74d2eb742841649bb4749b202407ca907a8eab342740f491ac815d1e0" exitCode=0
Sep 16 21:47:33.603712 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.603581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl" event={"ID":"6553a042-53fe-40f5-843c-4815273f5737","Type":"ContainerDied","Data":"0427ecb74d2eb742841649bb4749b202407ca907a8eab342740f491ac815d1e0"}
Sep 16 21:47:33.603712 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.603636    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl" event={"ID":"6553a042-53fe-40f5-843c-4815273f5737","Type":"ContainerDied","Data":"09e106c121ceafaa53fc2ef5de87fd38e44dbc91f6229dc365c9c3c0651d7dba"}
Sep 16 21:47:33.603712 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.603667    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09e106c121ceafaa53fc2ef5de87fd38e44dbc91f6229dc365c9c3c0651d7dba"
Sep 16 21:47:33.603712 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.603696    2563 scope.go:117] "RemoveContainer" containerID="d221954fca1b9ae36ffd19633c5fbfbc029df58cf6f3f4998ffa31580937f35f"
Sep 16 21:47:33.613250 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.613186    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:47:33.632880 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.632843    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-workload-certs\") pod \"6553a042-53fe-40f5-843c-4815273f5737\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") "
Sep 16 21:47:33.632880 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.632904    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-credential-socket\") pod \"6553a042-53fe-40f5-843c-4815273f5737\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") "
Sep 16 21:47:33.634308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.632940    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6553a042-53fe-40f5-843c-4815273f5737-istiod-ca-cert\") pod \"6553a042-53fe-40f5-843c-4815273f5737\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") "
Sep 16 21:47:33.634308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.632988    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6553a042-53fe-40f5-843c-4815273f5737-istio-token\") pod \"6553a042-53fe-40f5-843c-4815273f5737\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") "
Sep 16 21:47:33.634308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.633013    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6553a042-53fe-40f5-843c-4815273f5737-istio-podinfo\") pod \"6553a042-53fe-40f5-843c-4815273f5737\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") "
Sep 16 21:47:33.634308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.633087    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-workload-socket\") pod \"6553a042-53fe-40f5-843c-4815273f5737\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") "
Sep 16 21:47:33.634308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.633122    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-istio-data\") pod \"6553a042-53fe-40f5-843c-4815273f5737\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") "
Sep 16 21:47:33.634308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.633171    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-istio-envoy\") pod \"6553a042-53fe-40f5-843c-4815273f5737\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") "
Sep 16 21:47:33.634308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.633240    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6553a042-53fe-40f5-843c-4815273f5737" (UID: "6553a042-53fe-40f5-843c-4815273f5737"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:33.634308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.633282    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6553a042-53fe-40f5-843c-4815273f5737" (UID: "6553a042-53fe-40f5-843c-4815273f5737"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:33.634308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.633615    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-istio-data" (OuterVolumeSpecName: "istio-data") pod "6553a042-53fe-40f5-843c-4815273f5737" (UID: "6553a042-53fe-40f5-843c-4815273f5737"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:33.634308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.633717    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6553a042-53fe-40f5-843c-4815273f5737-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6553a042-53fe-40f5-843c-4815273f5737" (UID: "6553a042-53fe-40f5-843c-4815273f5737"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:47:33.634308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.633277    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sh57\" (UniqueName: \"kubernetes.io/projected/6553a042-53fe-40f5-843c-4815273f5737-kube-api-access-2sh57\") pod \"6553a042-53fe-40f5-843c-4815273f5737\" (UID: \"6553a042-53fe-40f5-843c-4815273f5737\") "
Sep 16 21:47:33.634308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.633849    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6553a042-53fe-40f5-843c-4815273f5737" (UID: "6553a042-53fe-40f5-843c-4815273f5737"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:33.635129 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.634517    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:33.635129 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.634538    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:33.635129 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.634551    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6553a042-53fe-40f5-843c-4815273f5737-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:33.635129 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.634565    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:33.635129 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.634578    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:33.635835 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.635809    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6553a042-53fe-40f5-843c-4815273f5737-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6553a042-53fe-40f5-843c-4815273f5737" (UID: "6553a042-53fe-40f5-843c-4815273f5737"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:47:33.636404 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.636383    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6553a042-53fe-40f5-843c-4815273f5737" (UID: "6553a042-53fe-40f5-843c-4815273f5737"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:33.637157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.637129    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6553a042-53fe-40f5-843c-4815273f5737-istio-token" (OuterVolumeSpecName: "istio-token") pod "6553a042-53fe-40f5-843c-4815273f5737" (UID: "6553a042-53fe-40f5-843c-4815273f5737"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:33.637268 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.637167    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6553a042-53fe-40f5-843c-4815273f5737-kube-api-access-2sh57" (OuterVolumeSpecName: "kube-api-access-2sh57") pod "6553a042-53fe-40f5-843c-4815273f5737" (UID: "6553a042-53fe-40f5-843c-4815273f5737"). InnerVolumeSpecName "kube-api-access-2sh57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:33.735039 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.735008    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6553a042-53fe-40f5-843c-4815273f5737-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:33.735039 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.735036    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6553a042-53fe-40f5-843c-4815273f5737-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:33.735039 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.735044    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6553a042-53fe-40f5-843c-4815273f5737-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:33.735274 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:33.735054    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sh57\" (UniqueName: \"kubernetes.io/projected/6553a042-53fe-40f5-843c-4815273f5737-kube-api-access-2sh57\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:34.611135 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:34.611101    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"
Sep 16 21:47:34.646147 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:34.646106    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"]
Sep 16 21:47:34.650201 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:34.650165    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-365ea1-openshift-default-5f8f9f85c8-phghl"]
Sep 16 21:47:35.585647 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.585606    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp" podUID="3d0500ae-a666-4950-be56-98850da5f370" containerName="main" containerID="cri-o://8d600fee723b2f3ee3f5adfc3a569f16082577eb44c081379192d16bdfa96170" gracePeriod=45
Sep 16 21:47:35.844276 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.844187    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:47:35.856597 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.856567    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-model-cache\") pod \"3d0500ae-a666-4950-be56-98850da5f370\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") "
Sep 16 21:47:35.856799 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.856617    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-dshm\") pod \"3d0500ae-a666-4950-be56-98850da5f370\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") "
Sep 16 21:47:35.856799 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.856680    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-home\") pod \"3d0500ae-a666-4950-be56-98850da5f370\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") "
Sep 16 21:47:35.856799 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.856734    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0500ae-a666-4950-be56-98850da5f370-tls-certs\") pod \"3d0500ae-a666-4950-be56-98850da5f370\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") "
Sep 16 21:47:35.856799 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.856779    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmglp\" (UniqueName: \"kubernetes.io/projected/3d0500ae-a666-4950-be56-98850da5f370-kube-api-access-gmglp\") pod \"3d0500ae-a666-4950-be56-98850da5f370\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") "
Sep 16 21:47:35.857001 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.856817    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-tmp-dir\") pod \"3d0500ae-a666-4950-be56-98850da5f370\" (UID: \"3d0500ae-a666-4950-be56-98850da5f370\") "
Sep 16 21:47:35.857001 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.856927    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d0500ae-a666-4950-be56-98850da5f370" (UID: "3d0500ae-a666-4950-be56-98850da5f370"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:35.857001 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.856942    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-home" (OuterVolumeSpecName: "home") pod "3d0500ae-a666-4950-be56-98850da5f370" (UID: "3d0500ae-a666-4950-be56-98850da5f370"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:35.857185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.857153    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d0500ae-a666-4950-be56-98850da5f370" (UID: "3d0500ae-a666-4950-be56-98850da5f370"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:35.857235 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.857194    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-model-cache\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:35.857235 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.857221    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-home\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:35.859050 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.859027    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-dshm" (OuterVolumeSpecName: "dshm") pod "3d0500ae-a666-4950-be56-98850da5f370" (UID: "3d0500ae-a666-4950-be56-98850da5f370"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:35.859151 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.859129    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d0500ae-a666-4950-be56-98850da5f370-kube-api-access-gmglp" (OuterVolumeSpecName: "kube-api-access-gmglp") pod "3d0500ae-a666-4950-be56-98850da5f370" (UID: "3d0500ae-a666-4950-be56-98850da5f370"). InnerVolumeSpecName "kube-api-access-gmglp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:35.859151 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.859135    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d0500ae-a666-4950-be56-98850da5f370-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d0500ae-a666-4950-be56-98850da5f370" (UID: "3d0500ae-a666-4950-be56-98850da5f370"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:47:35.958282 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.958236    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmglp\" (UniqueName: \"kubernetes.io/projected/3d0500ae-a666-4950-be56-98850da5f370-kube-api-access-gmglp\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:35.958282 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.958285    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-tmp-dir\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:35.958491 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.958301    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d0500ae-a666-4950-be56-98850da5f370-dshm\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:35.958491 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:35.958315    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d0500ae-a666-4950-be56-98850da5f370-tls-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:36.608637 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:36.608589    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6553a042-53fe-40f5-843c-4815273f5737" path="/var/lib/kubelet/pods/6553a042-53fe-40f5-843c-4815273f5737/volumes"
Sep 16 21:47:36.622679 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:36.622627    2563 generic.go:356] "Generic (PLEG): container finished" podID="3d0500ae-a666-4950-be56-98850da5f370" containerID="8d600fee723b2f3ee3f5adfc3a569f16082577eb44c081379192d16bdfa96170" exitCode=0
Sep 16 21:47:36.622891 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:36.622689    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"
Sep 16 21:47:36.622891 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:36.622692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp" event={"ID":"3d0500ae-a666-4950-be56-98850da5f370","Type":"ContainerDied","Data":"8d600fee723b2f3ee3f5adfc3a569f16082577eb44c081379192d16bdfa96170"}
Sep 16 21:47:36.622891 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:36.622751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp" event={"ID":"3d0500ae-a666-4950-be56-98850da5f370","Type":"ContainerDied","Data":"35ca525ada548030c4e6353f1b5396302798ef68c6bb7edc1e46ea6d0574437d"}
Sep 16 21:47:36.622891 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:36.622774    2563 scope.go:117] "RemoveContainer" containerID="8d600fee723b2f3ee3f5adfc3a569f16082577eb44c081379192d16bdfa96170"
Sep 16 21:47:36.632100 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:36.632080    2563 scope.go:117] "RemoveContainer" containerID="8d600fee723b2f3ee3f5adfc3a569f16082577eb44c081379192d16bdfa96170"
Sep 16 21:47:36.632454 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:47:36.632428    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d600fee723b2f3ee3f5adfc3a569f16082577eb44c081379192d16bdfa96170\": container with ID starting with 8d600fee723b2f3ee3f5adfc3a569f16082577eb44c081379192d16bdfa96170 not found: ID does not exist" containerID="8d600fee723b2f3ee3f5adfc3a569f16082577eb44c081379192d16bdfa96170"
Sep 16 21:47:36.632506 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:36.632466    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d600fee723b2f3ee3f5adfc3a569f16082577eb44c081379192d16bdfa96170"} err="failed to get container status \"8d600fee723b2f3ee3f5adfc3a569f16082577eb44c081379192d16bdfa96170\": rpc error: code = NotFound desc = could not find container \"8d600fee723b2f3ee3f5adfc3a569f16082577eb44c081379192d16bdfa96170\": container with ID starting with 8d600fee723b2f3ee3f5adfc3a569f16082577eb44c081379192d16bdfa96170 not found: ID does not exist"
Sep 16 21:47:36.638584 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:36.638555    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"]
Sep 16 21:47:36.641732 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:36.641697    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e047d8/simulator-w5-e047d8-kserve-5778687c78-nt5lp"]
Sep 16 21:47:38.041153 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:38.041116    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"]
Sep 16 21:47:38.041682 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:38.041339    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d" podUID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerName="istio-proxy" containerID="cri-o://2332f673de92c63b590901419b464abfb6daadc53fc277d238b6e87b7cdc5cde" gracePeriod=30
Sep 16 21:47:38.608110 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:38.608077    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d0500ae-a666-4950-be56-98850da5f370" path="/var/lib/kubelet/pods/3d0500ae-a666-4950-be56-98850da5f370/volumes"
Sep 16 21:47:42.619486 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:42.619446    2563 patch_prober.go:28] interesting pod/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 21:47:42.619969 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:42.619504    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d" podUID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 21:47:43.296753 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.296726    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:47:43.331401 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.331281    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thltf\" (UniqueName: \"kubernetes.io/projected/c09e02b9-d226-419a-8a83-89efcd4825fc-kube-api-access-thltf\") pod \"c09e02b9-d226-419a-8a83-89efcd4825fc\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") "
Sep 16 21:47:43.331401 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.331342    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-podinfo\") pod \"c09e02b9-d226-419a-8a83-89efcd4825fc\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") "
Sep 16 21:47:43.331619 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.331442    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09e02b9-d226-419a-8a83-89efcd4825fc-istiod-ca-cert\") pod \"c09e02b9-d226-419a-8a83-89efcd4825fc\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") "
Sep 16 21:47:43.331619 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.331532    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-data\") pod \"c09e02b9-d226-419a-8a83-89efcd4825fc\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") "
Sep 16 21:47:43.331756 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.331633    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-credential-socket\") pod \"c09e02b9-d226-419a-8a83-89efcd4825fc\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") "
Sep 16 21:47:43.331756 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.331675    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-workload-socket\") pod \"c09e02b9-d226-419a-8a83-89efcd4825fc\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") "
Sep 16 21:47:43.331756 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.331711    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-envoy\") pod \"c09e02b9-d226-419a-8a83-89efcd4825fc\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") "
Sep 16 21:47:43.331920 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.331765    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-token\") pod \"c09e02b9-d226-419a-8a83-89efcd4825fc\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") "
Sep 16 21:47:43.331920 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.331805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-workload-certs\") pod \"c09e02b9-d226-419a-8a83-89efcd4825fc\" (UID: \"c09e02b9-d226-419a-8a83-89efcd4825fc\") "
Sep 16 21:47:43.331920 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.331849    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "c09e02b9-d226-419a-8a83-89efcd4825fc" (UID: "c09e02b9-d226-419a-8a83-89efcd4825fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:43.331920 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.331867    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c09e02b9-d226-419a-8a83-89efcd4825fc" (UID: "c09e02b9-d226-419a-8a83-89efcd4825fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:43.331920 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.331899    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c09e02b9-d226-419a-8a83-89efcd4825fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c09e02b9-d226-419a-8a83-89efcd4825fc" (UID: "c09e02b9-d226-419a-8a83-89efcd4825fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:47:43.332253 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.332090    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c09e02b9-d226-419a-8a83-89efcd4825fc" (UID: "c09e02b9-d226-419a-8a83-89efcd4825fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:43.332253 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.332198    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:43.332253 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.332213    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:43.332417 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.332267    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:43.332417 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.332284    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09e02b9-d226-419a-8a83-89efcd4825fc-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:43.332417 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.332293    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c09e02b9-d226-419a-8a83-89efcd4825fc" (UID: "c09e02b9-d226-419a-8a83-89efcd4825fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:43.333773 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.333745    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09e02b9-d226-419a-8a83-89efcd4825fc-kube-api-access-thltf" (OuterVolumeSpecName: "kube-api-access-thltf") pod "c09e02b9-d226-419a-8a83-89efcd4825fc" (UID: "c09e02b9-d226-419a-8a83-89efcd4825fc"). InnerVolumeSpecName "kube-api-access-thltf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:43.333908 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.333870    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c09e02b9-d226-419a-8a83-89efcd4825fc" (UID: "c09e02b9-d226-419a-8a83-89efcd4825fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:47:43.334139 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.334116    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "c09e02b9-d226-419a-8a83-89efcd4825fc" (UID: "c09e02b9-d226-419a-8a83-89efcd4825fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:43.334247 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.334141    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c09e02b9-d226-419a-8a83-89efcd4825fc" (UID: "c09e02b9-d226-419a-8a83-89efcd4825fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:47:43.432877 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.432838    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:43.432877 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.432870    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:43.432877 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.432879    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09e02b9-d226-419a-8a83-89efcd4825fc-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:43.432877 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.432888    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thltf\" (UniqueName: \"kubernetes.io/projected/c09e02b9-d226-419a-8a83-89efcd4825fc-kube-api-access-thltf\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:43.433157 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.432898    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09e02b9-d226-419a-8a83-89efcd4825fc-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:43.662674 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.662565    2563 generic.go:356] "Generic (PLEG): container finished" podID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerID="2332f673de92c63b590901419b464abfb6daadc53fc277d238b6e87b7cdc5cde" exitCode=0
Sep 16 21:47:43.662674 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.662617    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"
Sep 16 21:47:43.663116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.662684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d" event={"ID":"c09e02b9-d226-419a-8a83-89efcd4825fc","Type":"ContainerDied","Data":"2332f673de92c63b590901419b464abfb6daadc53fc277d238b6e87b7cdc5cde"}
Sep 16 21:47:43.663116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.662738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d" event={"ID":"c09e02b9-d226-419a-8a83-89efcd4825fc","Type":"ContainerDied","Data":"e7c98fe4936224edd18a5fd7952e92120f7c4ab6d1c233eaef32e8860b8f4248"}
Sep 16 21:47:43.663116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.662759    2563 scope.go:117] "RemoveContainer" containerID="2332f673de92c63b590901419b464abfb6daadc53fc277d238b6e87b7cdc5cde"
Sep 16 21:47:43.672388 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.672370    2563 scope.go:117] "RemoveContainer" containerID="dbe87a25bfccf494935ae45ada54facb48b8e66548430b4346d0272e7c1157fd"
Sep 16 21:47:43.681562 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.681540    2563 scope.go:117] "RemoveContainer" containerID="2332f673de92c63b590901419b464abfb6daadc53fc277d238b6e87b7cdc5cde"
Sep 16 21:47:43.681909 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:47:43.681887    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2332f673de92c63b590901419b464abfb6daadc53fc277d238b6e87b7cdc5cde\": container with ID starting with 2332f673de92c63b590901419b464abfb6daadc53fc277d238b6e87b7cdc5cde not found: ID does not exist" containerID="2332f673de92c63b590901419b464abfb6daadc53fc277d238b6e87b7cdc5cde"
Sep 16 21:47:43.681983 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.681917    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2332f673de92c63b590901419b464abfb6daadc53fc277d238b6e87b7cdc5cde"} err="failed to get container status \"2332f673de92c63b590901419b464abfb6daadc53fc277d238b6e87b7cdc5cde\": rpc error: code = NotFound desc = could not find container \"2332f673de92c63b590901419b464abfb6daadc53fc277d238b6e87b7cdc5cde\": container with ID starting with 2332f673de92c63b590901419b464abfb6daadc53fc277d238b6e87b7cdc5cde not found: ID does not exist"
Sep 16 21:47:43.681983 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.681936    2563 scope.go:117] "RemoveContainer" containerID="dbe87a25bfccf494935ae45ada54facb48b8e66548430b4346d0272e7c1157fd"
Sep 16 21:47:43.682246 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:47:43.682227    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbe87a25bfccf494935ae45ada54facb48b8e66548430b4346d0272e7c1157fd\": container with ID starting with dbe87a25bfccf494935ae45ada54facb48b8e66548430b4346d0272e7c1157fd not found: ID does not exist" containerID="dbe87a25bfccf494935ae45ada54facb48b8e66548430b4346d0272e7c1157fd"
Sep 16 21:47:43.682291 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.682253    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe87a25bfccf494935ae45ada54facb48b8e66548430b4346d0272e7c1157fd"} err="failed to get container status \"dbe87a25bfccf494935ae45ada54facb48b8e66548430b4346d0272e7c1157fd\": rpc error: code = NotFound desc = could not find container \"dbe87a25bfccf494935ae45ada54facb48b8e66548430b4346d0272e7c1157fd\": container with ID starting with dbe87a25bfccf494935ae45ada54facb48b8e66548430b4346d0272e7c1157fd not found: ID does not exist"
Sep 16 21:47:43.682766 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.682745    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"]
Sep 16 21:47:43.687084 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:43.687061    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e047d8-openshift-default-557df7456b-4qw6d"]
Sep 16 21:47:44.609325 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:44.609294    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c09e02b9-d226-419a-8a83-89efcd4825fc" path="/var/lib/kubelet/pods/c09e02b9-d226-419a-8a83-89efcd4825fc/volumes"
Sep 16 21:47:48.775279 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775243    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7375-df7f28"]
Sep 16 21:47:48.775733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775582    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6553a042-53fe-40f5-843c-4815273f5737" containerName="istio-proxy"
Sep 16 21:47:48.775733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775594    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6553a042-53fe-40f5-843c-4815273f5737" containerName="istio-proxy"
Sep 16 21:47:48.775733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775604    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5" containerName="revoke-keys"
Sep 16 21:47:48.775733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775609    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5" containerName="revoke-keys"
Sep 16 21:47:48.775733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775628    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerName="istio-proxy"
Sep 16 21:47:48.775733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775634    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerName="istio-proxy"
Sep 16 21:47:48.775733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775640    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" containerName="istio-proxy"
Sep 16 21:47:48.775733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775646    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" containerName="istio-proxy"
Sep 16 21:47:48.775733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775671    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6553a042-53fe-40f5-843c-4815273f5737" containerName="istio-proxy"
Sep 16 21:47:48.775733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775677    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6553a042-53fe-40f5-843c-4815273f5737" containerName="istio-proxy"
Sep 16 21:47:48.775733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775684    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d0500ae-a666-4950-be56-98850da5f370" containerName="main"
Sep 16 21:47:48.775733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775689    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d0500ae-a666-4950-be56-98850da5f370" containerName="main"
Sep 16 21:47:48.775733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775695    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" containerName="istio-proxy"
Sep 16 21:47:48.775733 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775701    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" containerName="istio-proxy"
Sep 16 21:47:48.776430 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775756    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6553a042-53fe-40f5-843c-4815273f5737" containerName="istio-proxy"
Sep 16 21:47:48.776430 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775764    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" containerName="istio-proxy"
Sep 16 21:47:48.776430 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775771    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d0500ae-a666-4950-be56-98850da5f370" containerName="main"
Sep 16 21:47:48.776430 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775778    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d7831d5-d49f-41fe-9e06-c3f8c1027b11" containerName="istio-proxy"
Sep 16 21:47:48.776430 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775784    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerName="istio-proxy"
Sep 16 21:47:48.776430 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775790    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ccbceb-b8a1-433e-aeb8-5aaa70b828e5" containerName="revoke-keys"
Sep 16 21:47:48.776430 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.775796    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerName="istio-proxy"
Sep 16 21:47:48.779986 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.779965    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-df7f28"
Sep 16 21:47:48.782119 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.782096    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8588c\""
Sep 16 21:47:48.786854 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.786827    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7375-df7f28"]
Sep 16 21:47:48.883521 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.883483    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lft9n\" (UniqueName: \"kubernetes.io/projected/6164ba14-b13b-4105-a0a1-9d8bdad4027a-kube-api-access-lft9n\") pod \"test-curl-7375-df7f28\" (UID: \"6164ba14-b13b-4105-a0a1-9d8bdad4027a\") " pod="opendatahub/test-curl-7375-df7f28"
Sep 16 21:47:48.984484 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.984442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lft9n\" (UniqueName: \"kubernetes.io/projected/6164ba14-b13b-4105-a0a1-9d8bdad4027a-kube-api-access-lft9n\") pod \"test-curl-7375-df7f28\" (UID: \"6164ba14-b13b-4105-a0a1-9d8bdad4027a\") " pod="opendatahub/test-curl-7375-df7f28"
Sep 16 21:47:48.994325 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:48.994288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lft9n\" (UniqueName: \"kubernetes.io/projected/6164ba14-b13b-4105-a0a1-9d8bdad4027a-kube-api-access-lft9n\") pod \"test-curl-7375-df7f28\" (UID: \"6164ba14-b13b-4105-a0a1-9d8bdad4027a\") " pod="opendatahub/test-curl-7375-df7f28"
Sep 16 21:47:49.090561 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:49.090469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-df7f28"
Sep 16 21:47:49.362397 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:49.362369    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7375-df7f28"]
Sep 16 21:47:49.364795 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:47:49.364753    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6164ba14_b13b_4105_a0a1_9d8bdad4027a.slice/crio-d79493635f52bdca7015c18638dddc84f8c9964982d3f5ab88e551097205e55a WatchSource:0}: Error finding container d79493635f52bdca7015c18638dddc84f8c9964982d3f5ab88e551097205e55a: Status 404 returned error can't find the container with id d79493635f52bdca7015c18638dddc84f8c9964982d3f5ab88e551097205e55a
Sep 16 21:47:49.699945 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:49.699910    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7375-df7f28" event={"ID":"6164ba14-b13b-4105-a0a1-9d8bdad4027a","Type":"ContainerStarted","Data":"e01c7b074fc0b551770b006cdd8a03c10da1137313373a08fa08c6ef69909a74"}
Sep 16 21:47:49.700179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:49.699954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7375-df7f28" event={"ID":"6164ba14-b13b-4105-a0a1-9d8bdad4027a","Type":"ContainerStarted","Data":"d79493635f52bdca7015c18638dddc84f8c9964982d3f5ab88e551097205e55a"}
Sep 16 21:47:49.715125 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:49.715066    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7375-df7f28" podStartSLOduration=1.715046653 podStartE2EDuration="1.715046653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:47:49.712739379 +0000 UTC m=+1529.612949383" watchObservedRunningTime="2026-09-16 21:47:49.715046653 +0000 UTC m=+1529.615256656"
Sep 16 21:47:50.322736 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:50.322700    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7375-df7f28"]
Sep 16 21:47:50.326888 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:50.326856    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7375-df7f28"]
Sep 16 21:47:50.705145 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:47:50.705084    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7375-df7f28" container="test-curl-7375-df7f28"
Sep 16 21:47:51.659732 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:51.659696    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"]
Sep 16 21:47:51.710092 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:51.710049    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7375-df7f28" podUID="6164ba14-b13b-4105-a0a1-9d8bdad4027a" containerName="test-curl-7375-df7f28" containerID="cri-o://e01c7b074fc0b551770b006cdd8a03c10da1137313373a08fa08c6ef69909a74" gracePeriod=2
Sep 16 21:47:53.875966 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:53.875936    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-df7f28"
Sep 16 21:47:53.931061 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:53.931024    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lft9n\" (UniqueName: \"kubernetes.io/projected/6164ba14-b13b-4105-a0a1-9d8bdad4027a-kube-api-access-lft9n\") pod \"6164ba14-b13b-4105-a0a1-9d8bdad4027a\" (UID: \"6164ba14-b13b-4105-a0a1-9d8bdad4027a\") "
Sep 16 21:47:53.933166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:53.933134    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6164ba14-b13b-4105-a0a1-9d8bdad4027a-kube-api-access-lft9n" (OuterVolumeSpecName: "kube-api-access-lft9n") pod "6164ba14-b13b-4105-a0a1-9d8bdad4027a" (UID: "6164ba14-b13b-4105-a0a1-9d8bdad4027a"). InnerVolumeSpecName "kube-api-access-lft9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:54.032160 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:54.032062    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lft9n\" (UniqueName: \"kubernetes.io/projected/6164ba14-b13b-4105-a0a1-9d8bdad4027a-kube-api-access-lft9n\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:47:54.607998 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:54.607962    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6164ba14-b13b-4105-a0a1-9d8bdad4027a" path="/var/lib/kubelet/pods/6164ba14-b13b-4105-a0a1-9d8bdad4027a/volumes"
Sep 16 21:47:54.727176 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:54.727138    2563 generic.go:356] "Generic (PLEG): container finished" podID="6164ba14-b13b-4105-a0a1-9d8bdad4027a" containerID="e01c7b074fc0b551770b006cdd8a03c10da1137313373a08fa08c6ef69909a74" exitCode=137
Sep 16 21:47:54.727176 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:54.727182    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-df7f28"
Sep 16 21:47:54.727418 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:54.727234    2563 scope.go:117] "RemoveContainer" containerID="e01c7b074fc0b551770b006cdd8a03c10da1137313373a08fa08c6ef69909a74"
Sep 16 21:47:54.735647 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:54.735623    2563 scope.go:117] "RemoveContainer" containerID="e01c7b074fc0b551770b006cdd8a03c10da1137313373a08fa08c6ef69909a74"
Sep 16 21:47:54.735999 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:47:54.735977    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01c7b074fc0b551770b006cdd8a03c10da1137313373a08fa08c6ef69909a74\": container with ID starting with e01c7b074fc0b551770b006cdd8a03c10da1137313373a08fa08c6ef69909a74 not found: ID does not exist" containerID="e01c7b074fc0b551770b006cdd8a03c10da1137313373a08fa08c6ef69909a74"
Sep 16 21:47:54.736078 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:47:54.736013    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01c7b074fc0b551770b006cdd8a03c10da1137313373a08fa08c6ef69909a74"} err="failed to get container status \"e01c7b074fc0b551770b006cdd8a03c10da1137313373a08fa08c6ef69909a74\": rpc error: code = NotFound desc = could not find container \"e01c7b074fc0b551770b006cdd8a03c10da1137313373a08fa08c6ef69909a74\": container with ID starting with e01c7b074fc0b551770b006cdd8a03c10da1137313373a08fa08c6ef69909a74 not found: ID does not exist"
Sep 16 21:48:06.682117 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:06.682068    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96" podUID="2816143a-03ad-414b-b842-fb6b4b76984c" containerName="main" containerID="cri-o://ab0571d31596589f3f38ee7a249014a5ce2da3ee80697a1385513a293ecbaf85" gracePeriod=45
Sep 16 21:48:06.928826 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:06.928801    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:48:07.061221 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.061130    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkvdz\" (UniqueName: \"kubernetes.io/projected/2816143a-03ad-414b-b842-fb6b4b76984c-kube-api-access-jkvdz\") pod \"2816143a-03ad-414b-b842-fb6b4b76984c\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") "
Sep 16 21:48:07.061221 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.061182    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2816143a-03ad-414b-b842-fb6b4b76984c-tls-certs\") pod \"2816143a-03ad-414b-b842-fb6b4b76984c\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") "
Sep 16 21:48:07.061221 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.061217    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-dshm\") pod \"2816143a-03ad-414b-b842-fb6b4b76984c\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") "
Sep 16 21:48:07.061589 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.061239    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-model-cache\") pod \"2816143a-03ad-414b-b842-fb6b4b76984c\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") "
Sep 16 21:48:07.061589 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.061263    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-tmp-dir\") pod \"2816143a-03ad-414b-b842-fb6b4b76984c\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") "
Sep 16 21:48:07.061589 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.061361    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-home\") pod \"2816143a-03ad-414b-b842-fb6b4b76984c\" (UID: \"2816143a-03ad-414b-b842-fb6b4b76984c\") "
Sep 16 21:48:07.061589 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.061562    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-model-cache" (OuterVolumeSpecName: "model-cache") pod "2816143a-03ad-414b-b842-fb6b4b76984c" (UID: "2816143a-03ad-414b-b842-fb6b4b76984c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:48:07.061816 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.061579    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2816143a-03ad-414b-b842-fb6b4b76984c" (UID: "2816143a-03ad-414b-b842-fb6b4b76984c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:48:07.061816 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.061684    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-home" (OuterVolumeSpecName: "home") pod "2816143a-03ad-414b-b842-fb6b4b76984c" (UID: "2816143a-03ad-414b-b842-fb6b4b76984c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:48:07.061816 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.061783    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-home\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:48:07.061816 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.061796    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-model-cache\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:48:07.061816 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.061807    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-tmp-dir\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:48:07.063561 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.063531    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2816143a-03ad-414b-b842-fb6b4b76984c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2816143a-03ad-414b-b842-fb6b4b76984c" (UID: "2816143a-03ad-414b-b842-fb6b4b76984c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:48:07.063865 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.063850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-dshm" (OuterVolumeSpecName: "dshm") pod "2816143a-03ad-414b-b842-fb6b4b76984c" (UID: "2816143a-03ad-414b-b842-fb6b4b76984c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:48:07.063922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.063872    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2816143a-03ad-414b-b842-fb6b4b76984c-kube-api-access-jkvdz" (OuterVolumeSpecName: "kube-api-access-jkvdz") pod "2816143a-03ad-414b-b842-fb6b4b76984c" (UID: "2816143a-03ad-414b-b842-fb6b4b76984c"). InnerVolumeSpecName "kube-api-access-jkvdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:48:07.162595 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.162552    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkvdz\" (UniqueName: \"kubernetes.io/projected/2816143a-03ad-414b-b842-fb6b4b76984c-kube-api-access-jkvdz\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:48:07.162595 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.162590    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2816143a-03ad-414b-b842-fb6b4b76984c-tls-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:48:07.162595 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.162599    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2816143a-03ad-414b-b842-fb6b4b76984c-dshm\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:48:07.799436 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.799403    2563 generic.go:356] "Generic (PLEG): container finished" podID="2816143a-03ad-414b-b842-fb6b4b76984c" containerID="ab0571d31596589f3f38ee7a249014a5ce2da3ee80697a1385513a293ecbaf85" exitCode=0
Sep 16 21:48:07.799934 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.799471    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"
Sep 16 21:48:07.799934 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.799499    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96" event={"ID":"2816143a-03ad-414b-b842-fb6b4b76984c","Type":"ContainerDied","Data":"ab0571d31596589f3f38ee7a249014a5ce2da3ee80697a1385513a293ecbaf85"}
Sep 16 21:48:07.799934 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.799552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96" event={"ID":"2816143a-03ad-414b-b842-fb6b4b76984c","Type":"ContainerDied","Data":"5c3996f666ee2b1a4a91a198d57468c001bdff846e6eb2002c6cf5037bfac868"}
Sep 16 21:48:07.799934 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.799576    2563 scope.go:117] "RemoveContainer" containerID="ab0571d31596589f3f38ee7a249014a5ce2da3ee80697a1385513a293ecbaf85"
Sep 16 21:48:07.808474 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.808453    2563 scope.go:117] "RemoveContainer" containerID="ab0571d31596589f3f38ee7a249014a5ce2da3ee80697a1385513a293ecbaf85"
Sep 16 21:48:07.808956 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:48:07.808774    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0571d31596589f3f38ee7a249014a5ce2da3ee80697a1385513a293ecbaf85\": container with ID starting with ab0571d31596589f3f38ee7a249014a5ce2da3ee80697a1385513a293ecbaf85 not found: ID does not exist" containerID="ab0571d31596589f3f38ee7a249014a5ce2da3ee80697a1385513a293ecbaf85"
Sep 16 21:48:07.808956 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.808810    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0571d31596589f3f38ee7a249014a5ce2da3ee80697a1385513a293ecbaf85"} err="failed to get container status \"ab0571d31596589f3f38ee7a249014a5ce2da3ee80697a1385513a293ecbaf85\": rpc error: code = NotFound desc = could not find container \"ab0571d31596589f3f38ee7a249014a5ce2da3ee80697a1385513a293ecbaf85\": container with ID starting with ab0571d31596589f3f38ee7a249014a5ce2da3ee80697a1385513a293ecbaf85 not found: ID does not exist"
Sep 16 21:48:07.822698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.822674    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"]
Sep 16 21:48:07.826374 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:07.826353    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/auth-test-model-a7df81-kserve-8467d686d7-czj96"]
Sep 16 21:48:08.608339 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:08.608307    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2816143a-03ad-414b-b842-fb6b4b76984c" path="/var/lib/kubelet/pods/2816143a-03ad-414b-b842-fb6b4b76984c/volumes"
Sep 16 21:48:20.917475 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:20.917440    2563 scope.go:117] "RemoveContainer" containerID="0427ecb74d2eb742841649bb4749b202407ca907a8eab342740f491ac815d1e0"
Sep 16 21:48:23.087454 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.087420    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"]
Sep 16 21:48:23.087861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.087770    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2816143a-03ad-414b-b842-fb6b4b76984c" containerName="main"
Sep 16 21:48:23.087861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.087781    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2816143a-03ad-414b-b842-fb6b4b76984c" containerName="main"
Sep 16 21:48:23.087861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.087798    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6164ba14-b13b-4105-a0a1-9d8bdad4027a" containerName="test-curl-7375-df7f28"
Sep 16 21:48:23.087861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.087804    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6164ba14-b13b-4105-a0a1-9d8bdad4027a" containerName="test-curl-7375-df7f28"
Sep 16 21:48:23.087861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.087820    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerName="istio-proxy"
Sep 16 21:48:23.087861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.087825    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c09e02b9-d226-419a-8a83-89efcd4825fc" containerName="istio-proxy"
Sep 16 21:48:23.088071 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.087881    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2816143a-03ad-414b-b842-fb6b4b76984c" containerName="main"
Sep 16 21:48:23.088071 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.087889    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6553a042-53fe-40f5-843c-4815273f5737" containerName="istio-proxy"
Sep 16 21:48:23.088071 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.087899    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6164ba14-b13b-4105-a0a1-9d8bdad4027a" containerName="test-curl-7375-df7f28"
Sep 16 21:48:23.090735 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.090716    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.093309 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.093289    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7df81\"/\"kube-root-ca.crt\""
Sep 16 21:48:23.094062 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.094040    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7df81\"/\"default-dockercfg-vvzqj\""
Sep 16 21:48:23.094212 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.094122    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7df81\"/\"openshift-service-ca.crt\""
Sep 16 21:48:23.094212 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.094184    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a7df81\"/\"sub-test-model-a7df81-kserve-self-signed-certs\""
Sep 16 21:48:23.101518 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.101493    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"]
Sep 16 21:48:23.110501 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.110470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-dshm\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.110647 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.110510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-model-cache\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.110647 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.110526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-tmp-dir\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.110647 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.110568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6wrd\" (UniqueName: \"kubernetes.io/projected/d745d3d5-93f4-4047-8f42-30a2a8102569-kube-api-access-n6wrd\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.110647 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.110618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d745d3d5-93f4-4047-8f42-30a2a8102569-tls-certs\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.110647 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.110634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-home\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.211897 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.211853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-dshm\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.212079 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.211908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-model-cache\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.212079 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.211933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-tmp-dir\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.212079 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.211960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6wrd\" (UniqueName: \"kubernetes.io/projected/d745d3d5-93f4-4047-8f42-30a2a8102569-kube-api-access-n6wrd\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.212192 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.212115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d745d3d5-93f4-4047-8f42-30a2a8102569-tls-certs\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.212192 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.212161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-home\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.212404 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.212378    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-model-cache\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.212483 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.212424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-tmp-dir\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.212618 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.212597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-home\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.214243 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.214229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-dshm\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.214586 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.214571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d745d3d5-93f4-4047-8f42-30a2a8102569-tls-certs\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.219848 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.219826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6wrd\" (UniqueName: \"kubernetes.io/projected/d745d3d5-93f4-4047-8f42-30a2a8102569-kube-api-access-n6wrd\") pod \"sub-test-model-a7df81-kserve-776556dbfc-bncdg\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") " pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.402422 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.402331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:23.674216 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.674188    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"]
Sep 16 21:48:23.676771 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:48:23.676743    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd745d3d5_93f4_4047_8f42_30a2a8102569.slice/crio-5fa05b29c6712e50427805a7121b17a242183cb5c1746a7c3f15069e84f30818 WatchSource:0}: Error finding container 5fa05b29c6712e50427805a7121b17a242183cb5c1746a7c3f15069e84f30818: Status 404 returned error can't find the container with id 5fa05b29c6712e50427805a7121b17a242183cb5c1746a7c3f15069e84f30818
Sep 16 21:48:23.894408 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.894369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg" event={"ID":"d745d3d5-93f4-4047-8f42-30a2a8102569","Type":"ContainerStarted","Data":"585c9f614409f92eec7f2ed0498a64d86b4da4a82d64431f002aea838d8bef81"}
Sep 16 21:48:23.894607 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:23.894416    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg" event={"ID":"d745d3d5-93f4-4047-8f42-30a2a8102569","Type":"ContainerStarted","Data":"5fa05b29c6712e50427805a7121b17a242183cb5c1746a7c3f15069e84f30818"}
Sep 16 21:48:33.402963 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:33.402920    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:33.403373 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:33.402984    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:33.416044 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:33.416016    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:33.432127 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:33.432075    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg" podStartSLOduration=10.432058713 podStartE2EDuration="10.432058713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:48:23.915552101 +0000 UTC m=+1563.815762106" watchObservedRunningTime="2026-09-16 21:48:33.432058713 +0000 UTC m=+1573.332268716"
Sep 16 21:48:33.954131 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:33.954098    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:48:43.874899 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:43.874861    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"]
Sep 16 21:48:43.878907 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:43.878888    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:43.881213 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:43.881182    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1b68db\"/\"default-dockercfg-xb2m5\""
Sep 16 21:48:43.881346 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:43.881247    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1b68db\"/\"openshift-service-ca.crt\""
Sep 16 21:48:43.882164 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:43.882145    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1b68db\"/\"kube-root-ca.crt\""
Sep 16 21:48:43.882275 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:43.882254    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1b68db\"/\"sub-test-model-1b68db-kserve-self-signed-certs\""
Sep 16 21:48:43.888307 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:43.888284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"]
Sep 16 21:48:44.021902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.021866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmq4g\" (UniqueName: \"kubernetes.io/projected/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-kube-api-access-zmq4g\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.021902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.021905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-tls-certs\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.022127 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.022000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-model-cache\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.022127 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.022041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-dshm\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.022127 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.022091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-home\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.022127 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.022109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-tmp-dir\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.123321 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.123279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-model-cache\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.123507 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.123339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-dshm\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.123507 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.123372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-home\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.123507 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.123389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-tmp-dir\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.123507 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.123419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmq4g\" (UniqueName: \"kubernetes.io/projected/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-kube-api-access-zmq4g\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.123507 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.123438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-tls-certs\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.123807 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.123780    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-model-cache\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.123882 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.123790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-tmp-dir\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.123882 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.123843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-home\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.125954 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.125894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-dshm\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.126169 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.126149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-tls-certs\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.130902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.130882    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmq4g\" (UniqueName: \"kubernetes.io/projected/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-kube-api-access-zmq4g\") pod \"sub-test-model-1b68db-kserve-7c558f8877-z2mdw\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.190724 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.190682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:44.462210 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:44.462026    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"]
Sep 16 21:48:44.464998 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:48:44.464964    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe9b220e_fceb_4d88_9d98_2f63bf0755e7.slice/crio-1b0d7383dd9bebea8fe10ee08bb3c2f792ff5b26be5a40c015ee6979104604b4 WatchSource:0}: Error finding container 1b0d7383dd9bebea8fe10ee08bb3c2f792ff5b26be5a40c015ee6979104604b4: Status 404 returned error can't find the container with id 1b0d7383dd9bebea8fe10ee08bb3c2f792ff5b26be5a40c015ee6979104604b4
Sep 16 21:48:45.007729 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:45.007694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw" event={"ID":"fe9b220e-fceb-4d88-9d98-2f63bf0755e7","Type":"ContainerStarted","Data":"356adf1ab4da51d156b7f2e07d622f84b01cd636f712271c18598ea1dcef1bd0"}
Sep 16 21:48:45.008120 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:45.007738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw" event={"ID":"fe9b220e-fceb-4d88-9d98-2f63bf0755e7","Type":"ContainerStarted","Data":"1b0d7383dd9bebea8fe10ee08bb3c2f792ff5b26be5a40c015ee6979104604b4"}
Sep 16 21:48:45.023805 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:45.023747    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw" podStartSLOduration=2.023729475 podStartE2EDuration="2.023729475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:48:45.022782096 +0000 UTC m=+1584.922992100" watchObservedRunningTime="2026-09-16 21:48:45.023729475 +0000 UTC m=+1584.923939480"
Sep 16 21:48:54.191837 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:54.191794    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:54.192265 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:54.191855    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:54.204408 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:54.204376    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:48:55.070693 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:48:55.070647    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:49:11.402544 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:11.402510    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7375-a3cd31"]
Sep 16 21:49:11.406187 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:11.406169    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-a3cd31"
Sep 16 21:49:11.408813 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:11.408785    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8588c\""
Sep 16 21:49:11.409289 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:11.409267    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7375-a3cd31"]
Sep 16 21:49:11.481898 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:11.481863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7b4r\" (UniqueName: \"kubernetes.io/projected/8905b7f7-c0cc-449d-b51b-d4150f458c67-kube-api-access-v7b4r\") pod \"test-curl-7375-a3cd31\" (UID: \"8905b7f7-c0cc-449d-b51b-d4150f458c67\") " pod="opendatahub/test-curl-7375-a3cd31"
Sep 16 21:49:11.583035 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:11.583002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7b4r\" (UniqueName: \"kubernetes.io/projected/8905b7f7-c0cc-449d-b51b-d4150f458c67-kube-api-access-v7b4r\") pod \"test-curl-7375-a3cd31\" (UID: \"8905b7f7-c0cc-449d-b51b-d4150f458c67\") " pod="opendatahub/test-curl-7375-a3cd31"
Sep 16 21:49:11.591059 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:11.591038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7b4r\" (UniqueName: \"kubernetes.io/projected/8905b7f7-c0cc-449d-b51b-d4150f458c67-kube-api-access-v7b4r\") pod \"test-curl-7375-a3cd31\" (UID: \"8905b7f7-c0cc-449d-b51b-d4150f458c67\") " pod="opendatahub/test-curl-7375-a3cd31"
Sep 16 21:49:11.717055 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:11.717005    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-a3cd31"
Sep 16 21:49:11.987152 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:11.987126    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7375-a3cd31"]
Sep 16 21:49:11.989354 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:49:11.989327    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8905b7f7_c0cc_449d_b51b_d4150f458c67.slice/crio-f28367328228f7cfd0bf21ff11023d32a6243e0fa7157ab27de6dd283e7f2f77 WatchSource:0}: Error finding container f28367328228f7cfd0bf21ff11023d32a6243e0fa7157ab27de6dd283e7f2f77: Status 404 returned error can't find the container with id f28367328228f7cfd0bf21ff11023d32a6243e0fa7157ab27de6dd283e7f2f77
Sep 16 21:49:12.158382 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:12.158349    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7375-a3cd31" event={"ID":"8905b7f7-c0cc-449d-b51b-d4150f458c67","Type":"ContainerStarted","Data":"dba514b1627b1abffaea0699180950e7f397e3ff5d9cce021c62be7dc83d676a"}
Sep 16 21:49:12.158382 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:12.158390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7375-a3cd31" event={"ID":"8905b7f7-c0cc-449d-b51b-d4150f458c67","Type":"ContainerStarted","Data":"f28367328228f7cfd0bf21ff11023d32a6243e0fa7157ab27de6dd283e7f2f77"}
Sep 16 21:49:12.186290 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:12.186236    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7375-a3cd31" podStartSLOduration=1.186220125 podStartE2EDuration="1.186220125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:49:12.184005771 +0000 UTC m=+1612.084215774" watchObservedRunningTime="2026-09-16 21:49:12.186220125 +0000 UTC m=+1612.086430129"
Sep 16 21:49:12.753129 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:12.753090    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7375-a3cd31"]
Sep 16 21:49:12.757525 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:12.757494    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7375-a3cd31"]
Sep 16 21:49:12.863997 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:12.863963    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7375-220798"]
Sep 16 21:49:12.864303 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:12.864291    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8905b7f7-c0cc-449d-b51b-d4150f458c67" containerName="test-curl-7375-a3cd31"
Sep 16 21:49:12.864347 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:12.864305    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8905b7f7-c0cc-449d-b51b-d4150f458c67" containerName="test-curl-7375-a3cd31"
Sep 16 21:49:12.864382 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:12.864374    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8905b7f7-c0cc-449d-b51b-d4150f458c67" containerName="test-curl-7375-a3cd31"
Sep 16 21:49:12.867501 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:12.867485    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-220798"
Sep 16 21:49:12.873241 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:12.873213    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7375-220798"]
Sep 16 21:49:12.996864 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:12.996826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89m4w\" (UniqueName: \"kubernetes.io/projected/a3c705bf-657e-43ce-bba7-206a9a1eaf25-kube-api-access-89m4w\") pod \"test-curl-7375-220798\" (UID: \"a3c705bf-657e-43ce-bba7-206a9a1eaf25\") " pod="opendatahub/test-curl-7375-220798"
Sep 16 21:49:13.097845 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:13.097563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89m4w\" (UniqueName: \"kubernetes.io/projected/a3c705bf-657e-43ce-bba7-206a9a1eaf25-kube-api-access-89m4w\") pod \"test-curl-7375-220798\" (UID: \"a3c705bf-657e-43ce-bba7-206a9a1eaf25\") " pod="opendatahub/test-curl-7375-220798"
Sep 16 21:49:13.105867 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:13.105846    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89m4w\" (UniqueName: \"kubernetes.io/projected/a3c705bf-657e-43ce-bba7-206a9a1eaf25-kube-api-access-89m4w\") pod \"test-curl-7375-220798\" (UID: \"a3c705bf-657e-43ce-bba7-206a9a1eaf25\") " pod="opendatahub/test-curl-7375-220798"
Sep 16 21:49:13.163331 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:49:13.163303    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7375-a3cd31" container="test-curl-7375-a3cd31"
Sep 16 21:49:13.178208 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:13.178186    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-220798"
Sep 16 21:49:13.449442 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:13.449409    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7375-220798"]
Sep 16 21:49:13.451373 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:49:13.451348    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3c705bf_657e_43ce_bba7_206a9a1eaf25.slice/crio-00f97cc6d2d375d867ce05e6fe333af6b73b36b7ee7e83b6bf4e40bb9867272b WatchSource:0}: Error finding container 00f97cc6d2d375d867ce05e6fe333af6b73b36b7ee7e83b6bf4e40bb9867272b: Status 404 returned error can't find the container with id 00f97cc6d2d375d867ce05e6fe333af6b73b36b7ee7e83b6bf4e40bb9867272b
Sep 16 21:49:14.169977 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:14.169942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7375-220798" event={"ID":"a3c705bf-657e-43ce-bba7-206a9a1eaf25","Type":"ContainerStarted","Data":"b43c90a0ac61c5808b3f6014b916d866f3ef101c115d5998595e95a16dd16914"}
Sep 16 21:49:14.170415 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:14.169989    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7375-220798" event={"ID":"a3c705bf-657e-43ce-bba7-206a9a1eaf25","Type":"ContainerStarted","Data":"00f97cc6d2d375d867ce05e6fe333af6b73b36b7ee7e83b6bf4e40bb9867272b"}
Sep 16 21:49:14.170415 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:14.170011    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7375-a3cd31" podUID="8905b7f7-c0cc-449d-b51b-d4150f458c67" containerName="test-curl-7375-a3cd31" containerID="cri-o://dba514b1627b1abffaea0699180950e7f397e3ff5d9cce021c62be7dc83d676a" gracePeriod=2
Sep 16 21:49:14.171968 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:49:14.171947    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7375-a3cd31\" is forbidden: User \"system:node:ip-10-0-136-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-118.ec2.internal' and this object" podUID="8905b7f7-c0cc-449d-b51b-d4150f458c67" pod="opendatahub/test-curl-7375-a3cd31"
Sep 16 21:49:14.187415 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:49:14.187370    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7375-a3cd31\" is forbidden: User \"system:node:ip-10-0-136-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-118.ec2.internal' and this object" podUID="8905b7f7-c0cc-449d-b51b-d4150f458c67" pod="opendatahub/test-curl-7375-a3cd31"
Sep 16 21:49:14.188181 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:14.188146    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7375-220798" podStartSLOduration=2.188131395 podStartE2EDuration="2.188131395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:49:14.185887243 +0000 UTC m=+1614.086097247" watchObservedRunningTime="2026-09-16 21:49:14.188131395 +0000 UTC m=+1614.088341421"
Sep 16 21:49:14.753075 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:14.753024    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7375-220798"]
Sep 16 21:49:14.755573 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:14.755533    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7375-220798"]
Sep 16 21:49:15.175684 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:49:15.175644    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7375-220798" container="test-curl-7375-220798"
Sep 16 21:49:16.181105 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:16.181072    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7375-220798" podUID="a3c705bf-657e-43ce-bba7-206a9a1eaf25" containerName="test-curl-7375-220798" containerID="cri-o://b43c90a0ac61c5808b3f6014b916d866f3ef101c115d5998595e95a16dd16914" gracePeriod=2
Sep 16 21:49:16.182881 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:49:16.182846    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7375-220798\" is forbidden: User \"system:node:ip-10-0-136-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-118.ec2.internal' and this object" podUID="a3c705bf-657e-43ce-bba7-206a9a1eaf25" pod="opendatahub/test-curl-7375-220798"
Sep 16 21:49:16.348806 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:16.348781    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-a3cd31"
Sep 16 21:49:16.350508 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:49:16.350479    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7375-220798\" is forbidden: User \"system:node:ip-10-0-136-118.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-118.ec2.internal' and this object" podUID="a3c705bf-657e-43ce-bba7-206a9a1eaf25" pod="opendatahub/test-curl-7375-220798"
Sep 16 21:49:16.429152 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:16.429046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7b4r\" (UniqueName: \"kubernetes.io/projected/8905b7f7-c0cc-449d-b51b-d4150f458c67-kube-api-access-v7b4r\") pod \"8905b7f7-c0cc-449d-b51b-d4150f458c67\" (UID: \"8905b7f7-c0cc-449d-b51b-d4150f458c67\") "
Sep 16 21:49:16.431139 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:16.431104    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8905b7f7-c0cc-449d-b51b-d4150f458c67-kube-api-access-v7b4r" (OuterVolumeSpecName: "kube-api-access-v7b4r") pod "8905b7f7-c0cc-449d-b51b-d4150f458c67" (UID: "8905b7f7-c0cc-449d-b51b-d4150f458c67"). InnerVolumeSpecName "kube-api-access-v7b4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:49:16.530802 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:16.530769    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7b4r\" (UniqueName: \"kubernetes.io/projected/8905b7f7-c0cc-449d-b51b-d4150f458c67-kube-api-access-v7b4r\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:49:16.609457 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:16.609413    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8905b7f7-c0cc-449d-b51b-d4150f458c67" path="/var/lib/kubelet/pods/8905b7f7-c0cc-449d-b51b-d4150f458c67/volumes"
Sep 16 21:49:16.703835 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:16.703740    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"]
Sep 16 21:49:17.193295 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:17.193257    2563 generic.go:356] "Generic (PLEG): container finished" podID="8905b7f7-c0cc-449d-b51b-d4150f458c67" containerID="dba514b1627b1abffaea0699180950e7f397e3ff5d9cce021c62be7dc83d676a" exitCode=137
Sep 16 21:49:17.193825 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:17.193310    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-a3cd31"
Sep 16 21:49:17.193888 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:17.193348    2563 scope.go:117] "RemoveContainer" containerID="dba514b1627b1abffaea0699180950e7f397e3ff5d9cce021c62be7dc83d676a"
Sep 16 21:49:17.202874 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:17.202851    2563 scope.go:117] "RemoveContainer" containerID="dba514b1627b1abffaea0699180950e7f397e3ff5d9cce021c62be7dc83d676a"
Sep 16 21:49:17.203191 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:49:17.203170    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba514b1627b1abffaea0699180950e7f397e3ff5d9cce021c62be7dc83d676a\": container with ID starting with dba514b1627b1abffaea0699180950e7f397e3ff5d9cce021c62be7dc83d676a not found: ID does not exist" containerID="dba514b1627b1abffaea0699180950e7f397e3ff5d9cce021c62be7dc83d676a"
Sep 16 21:49:17.203251 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:17.203200    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba514b1627b1abffaea0699180950e7f397e3ff5d9cce021c62be7dc83d676a"} err="failed to get container status \"dba514b1627b1abffaea0699180950e7f397e3ff5d9cce021c62be7dc83d676a\": rpc error: code = NotFound desc = could not find container \"dba514b1627b1abffaea0699180950e7f397e3ff5d9cce021c62be7dc83d676a\": container with ID starting with dba514b1627b1abffaea0699180950e7f397e3ff5d9cce021c62be7dc83d676a not found: ID does not exist"
Sep 16 21:49:17.671323 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:17.671294    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"]
Sep 16 21:49:18.369698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:18.369671    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-220798"
Sep 16 21:49:18.450052 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:18.449959    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89m4w\" (UniqueName: \"kubernetes.io/projected/a3c705bf-657e-43ce-bba7-206a9a1eaf25-kube-api-access-89m4w\") pod \"a3c705bf-657e-43ce-bba7-206a9a1eaf25\" (UID: \"a3c705bf-657e-43ce-bba7-206a9a1eaf25\") "
Sep 16 21:49:18.452038 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:18.452016    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c705bf-657e-43ce-bba7-206a9a1eaf25-kube-api-access-89m4w" (OuterVolumeSpecName: "kube-api-access-89m4w") pod "a3c705bf-657e-43ce-bba7-206a9a1eaf25" (UID: "a3c705bf-657e-43ce-bba7-206a9a1eaf25"). InnerVolumeSpecName "kube-api-access-89m4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:49:18.551186 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:18.551150    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89m4w\" (UniqueName: \"kubernetes.io/projected/a3c705bf-657e-43ce-bba7-206a9a1eaf25-kube-api-access-89m4w\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:49:18.608055 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:18.608022    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3c705bf-657e-43ce-bba7-206a9a1eaf25" path="/var/lib/kubelet/pods/a3c705bf-657e-43ce-bba7-206a9a1eaf25/volumes"
Sep 16 21:49:19.206046 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:19.206014    2563 generic.go:356] "Generic (PLEG): container finished" podID="a3c705bf-657e-43ce-bba7-206a9a1eaf25" containerID="b43c90a0ac61c5808b3f6014b916d866f3ef101c115d5998595e95a16dd16914" exitCode=137
Sep 16 21:49:19.206258 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:19.206054    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7375-220798"
Sep 16 21:49:19.206258 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:19.206115    2563 scope.go:117] "RemoveContainer" containerID="b43c90a0ac61c5808b3f6014b916d866f3ef101c115d5998595e95a16dd16914"
Sep 16 21:49:19.215060 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:19.215040    2563 scope.go:117] "RemoveContainer" containerID="b43c90a0ac61c5808b3f6014b916d866f3ef101c115d5998595e95a16dd16914"
Sep 16 21:49:19.215397 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:49:19.215378    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b43c90a0ac61c5808b3f6014b916d866f3ef101c115d5998595e95a16dd16914\": container with ID starting with b43c90a0ac61c5808b3f6014b916d866f3ef101c115d5998595e95a16dd16914 not found: ID does not exist" containerID="b43c90a0ac61c5808b3f6014b916d866f3ef101c115d5998595e95a16dd16914"
Sep 16 21:49:19.215443 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:19.215409    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b43c90a0ac61c5808b3f6014b916d866f3ef101c115d5998595e95a16dd16914"} err="failed to get container status \"b43c90a0ac61c5808b3f6014b916d866f3ef101c115d5998595e95a16dd16914\": rpc error: code = NotFound desc = could not find container \"b43c90a0ac61c5808b3f6014b916d866f3ef101c115d5998595e95a16dd16914\": container with ID starting with b43c90a0ac61c5808b3f6014b916d866f3ef101c115d5998595e95a16dd16914 not found: ID does not exist"
Sep 16 21:49:31.722883 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:31.722833    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg" podUID="d745d3d5-93f4-4047-8f42-30a2a8102569" containerName="main" containerID="cri-o://585c9f614409f92eec7f2ed0498a64d86b4da4a82d64431f002aea838d8bef81" gracePeriod=45
Sep 16 21:49:31.976211 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:31.976143    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:49:32.088368 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.088330    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-tmp-dir\") pod \"d745d3d5-93f4-4047-8f42-30a2a8102569\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") "
Sep 16 21:49:32.088574 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.088393    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-model-cache\") pod \"d745d3d5-93f4-4047-8f42-30a2a8102569\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") "
Sep 16 21:49:32.088574 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.088413    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d745d3d5-93f4-4047-8f42-30a2a8102569-tls-certs\") pod \"d745d3d5-93f4-4047-8f42-30a2a8102569\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") "
Sep 16 21:49:32.088574 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.088435    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-dshm\") pod \"d745d3d5-93f4-4047-8f42-30a2a8102569\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") "
Sep 16 21:49:32.088574 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.088468    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6wrd\" (UniqueName: \"kubernetes.io/projected/d745d3d5-93f4-4047-8f42-30a2a8102569-kube-api-access-n6wrd\") pod \"d745d3d5-93f4-4047-8f42-30a2a8102569\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") "
Sep 16 21:49:32.088574 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.088510    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-home\") pod \"d745d3d5-93f4-4047-8f42-30a2a8102569\" (UID: \"d745d3d5-93f4-4047-8f42-30a2a8102569\") "
Sep 16 21:49:32.088889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.088711    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d745d3d5-93f4-4047-8f42-30a2a8102569" (UID: "d745d3d5-93f4-4047-8f42-30a2a8102569"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:49:32.088889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.088743    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-model-cache" (OuterVolumeSpecName: "model-cache") pod "d745d3d5-93f4-4047-8f42-30a2a8102569" (UID: "d745d3d5-93f4-4047-8f42-30a2a8102569"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:49:32.088889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.088861    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-home" (OuterVolumeSpecName: "home") pod "d745d3d5-93f4-4047-8f42-30a2a8102569" (UID: "d745d3d5-93f4-4047-8f42-30a2a8102569"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:49:32.089018 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.088908    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-tmp-dir\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:49:32.089018 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.088924    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-model-cache\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:49:32.090743 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.090707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d745d3d5-93f4-4047-8f42-30a2a8102569-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d745d3d5-93f4-4047-8f42-30a2a8102569" (UID: "d745d3d5-93f4-4047-8f42-30a2a8102569"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:49:32.090743 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.090738    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d745d3d5-93f4-4047-8f42-30a2a8102569-kube-api-access-n6wrd" (OuterVolumeSpecName: "kube-api-access-n6wrd") pod "d745d3d5-93f4-4047-8f42-30a2a8102569" (UID: "d745d3d5-93f4-4047-8f42-30a2a8102569"). InnerVolumeSpecName "kube-api-access-n6wrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:49:32.090889 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.090753    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-dshm" (OuterVolumeSpecName: "dshm") pod "d745d3d5-93f4-4047-8f42-30a2a8102569" (UID: "d745d3d5-93f4-4047-8f42-30a2a8102569"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:49:32.190420 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.190381    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d745d3d5-93f4-4047-8f42-30a2a8102569-tls-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:49:32.190420 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.190412    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-dshm\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:49:32.190420 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.190421    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6wrd\" (UniqueName: \"kubernetes.io/projected/d745d3d5-93f4-4047-8f42-30a2a8102569-kube-api-access-n6wrd\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:49:32.190420 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.190431    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d745d3d5-93f4-4047-8f42-30a2a8102569-home\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:49:32.280096 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.279997    2563 generic.go:356] "Generic (PLEG): container finished" podID="d745d3d5-93f4-4047-8f42-30a2a8102569" containerID="585c9f614409f92eec7f2ed0498a64d86b4da4a82d64431f002aea838d8bef81" exitCode=0
Sep 16 21:49:32.280096 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.280044    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"
Sep 16 21:49:32.280344 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.280101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg" event={"ID":"d745d3d5-93f4-4047-8f42-30a2a8102569","Type":"ContainerDied","Data":"585c9f614409f92eec7f2ed0498a64d86b4da4a82d64431f002aea838d8bef81"}
Sep 16 21:49:32.280344 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.280158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg" event={"ID":"d745d3d5-93f4-4047-8f42-30a2a8102569","Type":"ContainerDied","Data":"5fa05b29c6712e50427805a7121b17a242183cb5c1746a7c3f15069e84f30818"}
Sep 16 21:49:32.280344 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.280182    2563 scope.go:117] "RemoveContainer" containerID="585c9f614409f92eec7f2ed0498a64d86b4da4a82d64431f002aea838d8bef81"
Sep 16 21:49:32.289602 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.289582    2563 scope.go:117] "RemoveContainer" containerID="585c9f614409f92eec7f2ed0498a64d86b4da4a82d64431f002aea838d8bef81"
Sep 16 21:49:32.289902 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:49:32.289879    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585c9f614409f92eec7f2ed0498a64d86b4da4a82d64431f002aea838d8bef81\": container with ID starting with 585c9f614409f92eec7f2ed0498a64d86b4da4a82d64431f002aea838d8bef81 not found: ID does not exist" containerID="585c9f614409f92eec7f2ed0498a64d86b4da4a82d64431f002aea838d8bef81"
Sep 16 21:49:32.289972 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.289911    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585c9f614409f92eec7f2ed0498a64d86b4da4a82d64431f002aea838d8bef81"} err="failed to get container status \"585c9f614409f92eec7f2ed0498a64d86b4da4a82d64431f002aea838d8bef81\": rpc error: code = NotFound desc = could not find container \"585c9f614409f92eec7f2ed0498a64d86b4da4a82d64431f002aea838d8bef81\": container with ID starting with 585c9f614409f92eec7f2ed0498a64d86b4da4a82d64431f002aea838d8bef81 not found: ID does not exist"
Sep 16 21:49:32.300232 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.300207    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"]
Sep 16 21:49:32.304600 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.304580    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a7df81/sub-test-model-a7df81-kserve-776556dbfc-bncdg"]
Sep 16 21:49:32.608356 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.608274    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d745d3d5-93f4-4047-8f42-30a2a8102569" path="/var/lib/kubelet/pods/d745d3d5-93f4-4047-8f42-30a2a8102569/volumes"
Sep 16 21:49:32.691175 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.691133    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw" podUID="fe9b220e-fceb-4d88-9d98-2f63bf0755e7" containerName="main" containerID="cri-o://356adf1ab4da51d156b7f2e07d622f84b01cd636f712271c18598ea1dcef1bd0" gracePeriod=45
Sep 16 21:49:32.938843 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:32.938813    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:49:33.099021 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.098983    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-dshm\") pod \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") "
Sep 16 21:49:33.099231 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.099040    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-model-cache\") pod \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") "
Sep 16 21:49:33.099231 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.099062    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmq4g\" (UniqueName: \"kubernetes.io/projected/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-kube-api-access-zmq4g\") pod \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") "
Sep 16 21:49:33.099231 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.099083    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-tmp-dir\") pod \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") "
Sep 16 21:49:33.099231 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.099114    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-tls-certs\") pod \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") "
Sep 16 21:49:33.099231 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.099134    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-home\") pod \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\" (UID: \"fe9b220e-fceb-4d88-9d98-2f63bf0755e7\") "
Sep 16 21:49:33.099488 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.099313    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe9b220e-fceb-4d88-9d98-2f63bf0755e7" (UID: "fe9b220e-fceb-4d88-9d98-2f63bf0755e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:49:33.099488 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.099337    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe9b220e-fceb-4d88-9d98-2f63bf0755e7" (UID: "fe9b220e-fceb-4d88-9d98-2f63bf0755e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:49:33.099488 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.099463    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-home" (OuterVolumeSpecName: "home") pod "fe9b220e-fceb-4d88-9d98-2f63bf0755e7" (UID: "fe9b220e-fceb-4d88-9d98-2f63bf0755e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:49:33.099587 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.099514    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-model-cache\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:49:33.099587 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.099531    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-tmp-dir\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:49:33.101412 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.101380    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-dshm" (OuterVolumeSpecName: "dshm") pod "fe9b220e-fceb-4d88-9d98-2f63bf0755e7" (UID: "fe9b220e-fceb-4d88-9d98-2f63bf0755e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:49:33.101525 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.101391    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe9b220e-fceb-4d88-9d98-2f63bf0755e7" (UID: "fe9b220e-fceb-4d88-9d98-2f63bf0755e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:49:33.101525 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.101450    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-kube-api-access-zmq4g" (OuterVolumeSpecName: "kube-api-access-zmq4g") pod "fe9b220e-fceb-4d88-9d98-2f63bf0755e7" (UID: "fe9b220e-fceb-4d88-9d98-2f63bf0755e7"). InnerVolumeSpecName "kube-api-access-zmq4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:49:33.200882 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.200850    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-dshm\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:49:33.200882 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.200882    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmq4g\" (UniqueName: \"kubernetes.io/projected/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-kube-api-access-zmq4g\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:49:33.201081 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.200893    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-tls-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:49:33.201081 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.200902    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9b220e-fceb-4d88-9d98-2f63bf0755e7-home\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:49:33.287081 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.287045    2563 generic.go:356] "Generic (PLEG): container finished" podID="fe9b220e-fceb-4d88-9d98-2f63bf0755e7" containerID="356adf1ab4da51d156b7f2e07d622f84b01cd636f712271c18598ea1dcef1bd0" exitCode=0
Sep 16 21:49:33.287249 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.287095    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"
Sep 16 21:49:33.287249 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.287132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw" event={"ID":"fe9b220e-fceb-4d88-9d98-2f63bf0755e7","Type":"ContainerDied","Data":"356adf1ab4da51d156b7f2e07d622f84b01cd636f712271c18598ea1dcef1bd0"}
Sep 16 21:49:33.287249 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.287179    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw" event={"ID":"fe9b220e-fceb-4d88-9d98-2f63bf0755e7","Type":"ContainerDied","Data":"1b0d7383dd9bebea8fe10ee08bb3c2f792ff5b26be5a40c015ee6979104604b4"}
Sep 16 21:49:33.287249 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.287197    2563 scope.go:117] "RemoveContainer" containerID="356adf1ab4da51d156b7f2e07d622f84b01cd636f712271c18598ea1dcef1bd0"
Sep 16 21:49:33.296343 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.296325    2563 scope.go:117] "RemoveContainer" containerID="356adf1ab4da51d156b7f2e07d622f84b01cd636f712271c18598ea1dcef1bd0"
Sep 16 21:49:33.296600 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:49:33.296581    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"356adf1ab4da51d156b7f2e07d622f84b01cd636f712271c18598ea1dcef1bd0\": container with ID starting with 356adf1ab4da51d156b7f2e07d622f84b01cd636f712271c18598ea1dcef1bd0 not found: ID does not exist" containerID="356adf1ab4da51d156b7f2e07d622f84b01cd636f712271c18598ea1dcef1bd0"
Sep 16 21:49:33.296649 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.296608    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"356adf1ab4da51d156b7f2e07d622f84b01cd636f712271c18598ea1dcef1bd0"} err="failed to get container status \"356adf1ab4da51d156b7f2e07d622f84b01cd636f712271c18598ea1dcef1bd0\": rpc error: code = NotFound desc = could not find container \"356adf1ab4da51d156b7f2e07d622f84b01cd636f712271c18598ea1dcef1bd0\": container with ID starting with 356adf1ab4da51d156b7f2e07d622f84b01cd636f712271c18598ea1dcef1bd0 not found: ID does not exist"
Sep 16 21:49:33.307842 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.307816    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"]
Sep 16 21:49:33.311727 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:33.311705    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/sub-test-model-1b68db-kserve-7c558f8877-z2mdw"]
Sep 16 21:49:34.613634 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:49:34.609966    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe9b220e-fceb-4d88-9d98-2f63bf0755e7" path="/var/lib/kubelet/pods/fe9b220e-fceb-4d88-9d98-2f63bf0755e7/volumes"
Sep 16 21:50:09.279961 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.279927    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"]
Sep 16 21:50:09.280433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.280259    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d745d3d5-93f4-4047-8f42-30a2a8102569" containerName="main"
Sep 16 21:50:09.280433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.280270    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d745d3d5-93f4-4047-8f42-30a2a8102569" containerName="main"
Sep 16 21:50:09.280433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.280285    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe9b220e-fceb-4d88-9d98-2f63bf0755e7" containerName="main"
Sep 16 21:50:09.280433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.280291    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe9b220e-fceb-4d88-9d98-2f63bf0755e7" containerName="main"
Sep 16 21:50:09.280433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.280307    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3c705bf-657e-43ce-bba7-206a9a1eaf25" containerName="test-curl-7375-220798"
Sep 16 21:50:09.280433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.280312    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3c705bf-657e-43ce-bba7-206a9a1eaf25" containerName="test-curl-7375-220798"
Sep 16 21:50:09.280433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.280360    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3c705bf-657e-43ce-bba7-206a9a1eaf25" containerName="test-curl-7375-220798"
Sep 16 21:50:09.280433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.280369    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d745d3d5-93f4-4047-8f42-30a2a8102569" containerName="main"
Sep 16 21:50:09.280433 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.280376    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe9b220e-fceb-4d88-9d98-2f63bf0755e7" containerName="main"
Sep 16 21:50:09.291780 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.291751    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.293633 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.293586    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"]
Sep 16 21:50:09.294814 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.294790    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1b68db\"/\"kube-root-ca.crt\""
Sep 16 21:50:09.294962 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.294791    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1b68db\"/\"openshift-service-ca.crt\""
Sep 16 21:50:09.294962 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.294795    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1b68db\"/\"default-dockercfg-xb2m5\""
Sep 16 21:50:09.295090 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.294796    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1b68db\"/\"rate-test-model-1b68db-kserve-self-signed-certs\""
Sep 16 21:50:09.443636 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.443596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-dshm\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.443636 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.443644    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-model-cache\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.443960 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.443781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6clcc\" (UniqueName: \"kubernetes.io/projected/116efe38-bcef-4e0e-b1c6-d95fee218be9-kube-api-access-6clcc\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.443960 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.443827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-tmp-dir\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.443960 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.443891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-home\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.443960 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.443932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/116efe38-bcef-4e0e-b1c6-d95fee218be9-tls-certs\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.545518 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.545420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-model-cache\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.545518 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.545486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6clcc\" (UniqueName: \"kubernetes.io/projected/116efe38-bcef-4e0e-b1c6-d95fee218be9-kube-api-access-6clcc\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.545518 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.545516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-tmp-dir\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.545836 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.545555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-home\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.545836 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.545587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/116efe38-bcef-4e0e-b1c6-d95fee218be9-tls-certs\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.545836 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.545753    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-dshm\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.545991 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.545918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-model-cache\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.545991 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.545961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-home\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.546112 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.546005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-tmp-dir\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.547961 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.547934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-dshm\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.548185 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.548169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/116efe38-bcef-4e0e-b1c6-d95fee218be9-tls-certs\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.555516 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.555490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6clcc\" (UniqueName: \"kubernetes.io/projected/116efe38-bcef-4e0e-b1c6-d95fee218be9-kube-api-access-6clcc\") pod \"rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") " pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.603783 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.603745    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:09.885767 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:09.885742    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"]
Sep 16 21:50:09.888333 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:50:09.888301    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod116efe38_bcef_4e0e_b1c6_d95fee218be9.slice/crio-e7c07a4a0c4ba49c2ea2814cb325f5a7e74fdb5b15967c285e284593497cf89a WatchSource:0}: Error finding container e7c07a4a0c4ba49c2ea2814cb325f5a7e74fdb5b15967c285e284593497cf89a: Status 404 returned error can't find the container with id e7c07a4a0c4ba49c2ea2814cb325f5a7e74fdb5b15967c285e284593497cf89a
Sep 16 21:50:10.479411 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:10.479372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx" event={"ID":"116efe38-bcef-4e0e-b1c6-d95fee218be9","Type":"ContainerStarted","Data":"ec5ebeb50f8bf6eecf80fde9dedc5878e5d183e990377e3b6b0e40be94d7234e"}
Sep 16 21:50:10.479411 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:10.479412    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx" event={"ID":"116efe38-bcef-4e0e-b1c6-d95fee218be9","Type":"ContainerStarted","Data":"e7c07a4a0c4ba49c2ea2814cb325f5a7e74fdb5b15967c285e284593497cf89a"}
Sep 16 21:50:10.495158 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:10.495103    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx" podStartSLOduration=1.49508562 podStartE2EDuration="1.49508562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:50:10.494353027 +0000 UTC m=+1670.394563030" watchObservedRunningTime="2026-09-16 21:50:10.49508562 +0000 UTC m=+1670.395295632"
Sep 16 21:50:19.603902 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:19.603864    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:19.604350 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:19.604142    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:19.616908 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:19.616881    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:50:20.549011 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:50:20.548982    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:51:05.994087 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:05.993998    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"]
Sep 16 21:51:06.701402 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.701368    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"]
Sep 16 21:51:06.704941 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.704920    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.707406 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.707385    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ad0222-openshift-default-dockercfg-k7mmh\""
Sep 16 21:51:06.715939 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.715903    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"]
Sep 16 21:51:06.797564 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.797526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-istio-data\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.797796 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.797585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-workload-certs\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.797796 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.797639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57903415-58fe-452e-ae2a-eb30704bcff0-istiod-ca-cert\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.797796 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.797742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-credential-socket\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.797796 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.797782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-istio-envoy\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.798032 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.797843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-workload-socket\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.798032 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.797891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57903415-58fe-452e-ae2a-eb30704bcff0-istio-podinfo\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.798032 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.797927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gcmv\" (UniqueName: \"kubernetes.io/projected/57903415-58fe-452e-ae2a-eb30704bcff0-kube-api-access-4gcmv\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.798032 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.797980    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57903415-58fe-452e-ae2a-eb30704bcff0-istio-token\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.899095 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.899057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57903415-58fe-452e-ae2a-eb30704bcff0-istio-token\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.899289 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.899111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-istio-data\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.899289 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.899137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-workload-certs\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.899289 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.899164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57903415-58fe-452e-ae2a-eb30704bcff0-istiod-ca-cert\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.899289 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.899222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-credential-socket\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.899289 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.899258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-istio-envoy\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.899289 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.899288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-workload-socket\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.899623 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.899374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57903415-58fe-452e-ae2a-eb30704bcff0-istio-podinfo\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.899623 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.899415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcmv\" (UniqueName: \"kubernetes.io/projected/57903415-58fe-452e-ae2a-eb30704bcff0-kube-api-access-4gcmv\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.899623 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.899557    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-istio-data\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.899786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.899637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-workload-certs\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.899786 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.899737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-workload-socket\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.899910 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.899797    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-credential-socket\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.900028 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.900005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57903415-58fe-452e-ae2a-eb30704bcff0-istiod-ca-cert\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.901785 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.901757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-istio-envoy\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.901976 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.901958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57903415-58fe-452e-ae2a-eb30704bcff0-istio-podinfo\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.906509 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.906488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57903415-58fe-452e-ae2a-eb30704bcff0-istio-token\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:06.906705 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:06.906687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gcmv\" (UniqueName: \"kubernetes.io/projected/57903415-58fe-452e-ae2a-eb30704bcff0-kube-api-access-4gcmv\") pod \"e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") " pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:07.016328 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:07.016229    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:07.290896 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:07.290869    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"]
Sep 16 21:51:07.292750 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:51:07.292721    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57903415_58fe_452e_ae2a_eb30704bcff0.slice/crio-75277ce60a8351b4d5fc84da2830b7703173a84954d1cc636fe3d2b6bfb42449 WatchSource:0}: Error finding container 75277ce60a8351b4d5fc84da2830b7703173a84954d1cc636fe3d2b6bfb42449: Status 404 returned error can't find the container with id 75277ce60a8351b4d5fc84da2830b7703173a84954d1cc636fe3d2b6bfb42449
Sep 16 21:51:07.295071 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:07.295039    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:51:07.295146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:07.295108    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:51:07.295146 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:07.295136    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:51:07.772138 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:07.772102    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8" event={"ID":"57903415-58fe-452e-ae2a-eb30704bcff0","Type":"ContainerStarted","Data":"5a9b340cb02cb4f2a0d2a95f3cbeb47ee4cd22be52adeaec0b82a7e4f602e559"}
Sep 16 21:51:07.772354 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:07.772166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8" event={"ID":"57903415-58fe-452e-ae2a-eb30704bcff0","Type":"ContainerStarted","Data":"75277ce60a8351b4d5fc84da2830b7703173a84954d1cc636fe3d2b6bfb42449"}
Sep 16 21:51:07.789795 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:07.789739    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8" podStartSLOduration=1.789720917 podStartE2EDuration="1.789720917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:51:07.788955302 +0000 UTC m=+1727.689165304" watchObservedRunningTime="2026-09-16 21:51:07.789720917 +0000 UTC m=+1727.689930922"
Sep 16 21:51:08.016767 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:08.016734    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:08.021621 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:08.021597    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:08.777276 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:08.777249    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:08.778385 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:08.778370    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:20.987116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:20.987086    2563 scope.go:117] "RemoveContainer" containerID="788fd3d50bc2aa87041456e65e7f0be1fb7dee30caebe97f56c685272581f679"
Sep 16 21:51:20.995241 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:20.995221    2563 scope.go:117] "RemoveContainer" containerID="bd05e6e5be70d4fe040f18dac6a23efb9d9a584a94ddb28fe3efe8301b6a1cb0"
Sep 16 21:51:21.013297 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.013265    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx" podUID="116efe38-bcef-4e0e-b1c6-d95fee218be9" containerName="main" containerID="cri-o://ec5ebeb50f8bf6eecf80fde9dedc5878e5d183e990377e3b6b0e40be94d7234e" gracePeriod=45
Sep 16 21:51:21.253989 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.253967    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:51:21.343787 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.343756    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-dshm\") pod \"116efe38-bcef-4e0e-b1c6-d95fee218be9\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") "
Sep 16 21:51:21.343787 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.343787    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-tmp-dir\") pod \"116efe38-bcef-4e0e-b1c6-d95fee218be9\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") "
Sep 16 21:51:21.344003 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.343806    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-home\") pod \"116efe38-bcef-4e0e-b1c6-d95fee218be9\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") "
Sep 16 21:51:21.344003 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.343833    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6clcc\" (UniqueName: \"kubernetes.io/projected/116efe38-bcef-4e0e-b1c6-d95fee218be9-kube-api-access-6clcc\") pod \"116efe38-bcef-4e0e-b1c6-d95fee218be9\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") "
Sep 16 21:51:21.344003 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.343935    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-model-cache\") pod \"116efe38-bcef-4e0e-b1c6-d95fee218be9\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") "
Sep 16 21:51:21.344003 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.343996    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/116efe38-bcef-4e0e-b1c6-d95fee218be9-tls-certs\") pod \"116efe38-bcef-4e0e-b1c6-d95fee218be9\" (UID: \"116efe38-bcef-4e0e-b1c6-d95fee218be9\") "
Sep 16 21:51:21.344213 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.344043    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "116efe38-bcef-4e0e-b1c6-d95fee218be9" (UID: "116efe38-bcef-4e0e-b1c6-d95fee218be9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:51:21.344213 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.344057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-home" (OuterVolumeSpecName: "home") pod "116efe38-bcef-4e0e-b1c6-d95fee218be9" (UID: "116efe38-bcef-4e0e-b1c6-d95fee218be9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:51:21.344319 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.344219    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-model-cache" (OuterVolumeSpecName: "model-cache") pod "116efe38-bcef-4e0e-b1c6-d95fee218be9" (UID: "116efe38-bcef-4e0e-b1c6-d95fee218be9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:51:21.344371 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.344350    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-tmp-dir\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:51:21.344418 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.344369    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-home\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:51:21.344418 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.344385    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-model-cache\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:51:21.346136 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.346103    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116efe38-bcef-4e0e-b1c6-d95fee218be9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "116efe38-bcef-4e0e-b1c6-d95fee218be9" (UID: "116efe38-bcef-4e0e-b1c6-d95fee218be9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:51:21.346260 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.346117    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116efe38-bcef-4e0e-b1c6-d95fee218be9-kube-api-access-6clcc" (OuterVolumeSpecName: "kube-api-access-6clcc") pod "116efe38-bcef-4e0e-b1c6-d95fee218be9" (UID: "116efe38-bcef-4e0e-b1c6-d95fee218be9"). InnerVolumeSpecName "kube-api-access-6clcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:51:21.346260 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.346185    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-dshm" (OuterVolumeSpecName: "dshm") pod "116efe38-bcef-4e0e-b1c6-d95fee218be9" (UID: "116efe38-bcef-4e0e-b1c6-d95fee218be9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:51:21.444907 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.444876    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/116efe38-bcef-4e0e-b1c6-d95fee218be9-tls-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:51:21.444907 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.444900    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/116efe38-bcef-4e0e-b1c6-d95fee218be9-dshm\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:51:21.444907 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.444910    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6clcc\" (UniqueName: \"kubernetes.io/projected/116efe38-bcef-4e0e-b1c6-d95fee218be9-kube-api-access-6clcc\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:51:21.851560 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.851516    2563 generic.go:356] "Generic (PLEG): container finished" podID="116efe38-bcef-4e0e-b1c6-d95fee218be9" containerID="ec5ebeb50f8bf6eecf80fde9dedc5878e5d183e990377e3b6b0e40be94d7234e" exitCode=0
Sep 16 21:51:21.851805 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.851567    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"
Sep 16 21:51:21.851805 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.851610    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx" event={"ID":"116efe38-bcef-4e0e-b1c6-d95fee218be9","Type":"ContainerDied","Data":"ec5ebeb50f8bf6eecf80fde9dedc5878e5d183e990377e3b6b0e40be94d7234e"}
Sep 16 21:51:21.851805 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.851674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx" event={"ID":"116efe38-bcef-4e0e-b1c6-d95fee218be9","Type":"ContainerDied","Data":"e7c07a4a0c4ba49c2ea2814cb325f5a7e74fdb5b15967c285e284593497cf89a"}
Sep 16 21:51:21.851805 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.851703    2563 scope.go:117] "RemoveContainer" containerID="ec5ebeb50f8bf6eecf80fde9dedc5878e5d183e990377e3b6b0e40be94d7234e"
Sep 16 21:51:21.863629 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.863609    2563 scope.go:117] "RemoveContainer" containerID="ec5ebeb50f8bf6eecf80fde9dedc5878e5d183e990377e3b6b0e40be94d7234e"
Sep 16 21:51:21.863951 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:51:21.863932    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5ebeb50f8bf6eecf80fde9dedc5878e5d183e990377e3b6b0e40be94d7234e\": container with ID starting with ec5ebeb50f8bf6eecf80fde9dedc5878e5d183e990377e3b6b0e40be94d7234e not found: ID does not exist" containerID="ec5ebeb50f8bf6eecf80fde9dedc5878e5d183e990377e3b6b0e40be94d7234e"
Sep 16 21:51:21.864023 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.863960    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5ebeb50f8bf6eecf80fde9dedc5878e5d183e990377e3b6b0e40be94d7234e"} err="failed to get container status \"ec5ebeb50f8bf6eecf80fde9dedc5878e5d183e990377e3b6b0e40be94d7234e\": rpc error: code = NotFound desc = could not find container \"ec5ebeb50f8bf6eecf80fde9dedc5878e5d183e990377e3b6b0e40be94d7234e\": container with ID starting with ec5ebeb50f8bf6eecf80fde9dedc5878e5d183e990377e3b6b0e40be94d7234e not found: ID does not exist"
Sep 16 21:51:21.874078 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.874056    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"]
Sep 16 21:51:21.877921 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:21.877900    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1b68db/rate-test-model-1b68db-kserve-5b56cdfddc-rv8tx"]
Sep 16 21:51:22.608142 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:22.608099    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="116efe38-bcef-4e0e-b1c6-d95fee218be9" path="/var/lib/kubelet/pods/116efe38-bcef-4e0e-b1c6-d95fee218be9/volumes"
Sep 16 21:51:35.593860 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.593825    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"]
Sep 16 21:51:35.596452 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.594360    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="116efe38-bcef-4e0e-b1c6-d95fee218be9" containerName="main"
Sep 16 21:51:35.596452 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.594376    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="116efe38-bcef-4e0e-b1c6-d95fee218be9" containerName="main"
Sep 16 21:51:35.596452 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.594471    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="116efe38-bcef-4e0e-b1c6-d95fee218be9" containerName="main"
Sep 16 21:51:35.605893 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.605867    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.608925 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.608902    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-03eca6-openshift-default-dockercfg-dkq8h\""
Sep 16 21:51:35.610678 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.610637    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"]
Sep 16 21:51:35.683608 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.683557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-workload-certs\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.683861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.683620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-token\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.683861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.683762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-workload-socket\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.683861 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.683794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-data\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.684056 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.683987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-credential-socket\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.684201 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.684179    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcjhh\" (UniqueName: \"kubernetes.io/projected/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-kube-api-access-mcjhh\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.684289 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.684272    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istiod-ca-cert\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.684354 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.684335    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-podinfo\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.684407 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.684389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-envoy\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.785320 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.785279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istiod-ca-cert\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.785669 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.785626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-podinfo\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.786302 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.786280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-envoy\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.786489 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.786468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-workload-certs\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.786922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.786905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-token\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.787089 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.787074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-workload-socket\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.787481 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.787464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-data\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.787634 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.787618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-credential-socket\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.787817 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.787792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcjhh\" (UniqueName: \"kubernetes.io/projected/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-kube-api-access-mcjhh\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.788249 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.786856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-workload-certs\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.788632 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.788611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-data\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.788793 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.787421    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-workload-socket\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.789148 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.789128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-credential-socket\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.789261 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.786933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istiod-ca-cert\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.791564 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.791521    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-podinfo\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.791896 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.791872    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-envoy\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.796697 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.796630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-token\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.796961 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.796942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcjhh\" (UniqueName: \"kubernetes.io/projected/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-kube-api-access-mcjhh\") pod \"e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") " pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:35.920571 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:35.920547    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:36.220704 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:36.220677    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"]
Sep 16 21:51:36.222924 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:51:36.222883    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6978ea7c_6aae_4c24_adf9_6a1498e9aabc.slice/crio-c7b80b92b9cd457ece139962a3cddaf99f60e167ced8c8107a1ea53983fd6a57 WatchSource:0}: Error finding container c7b80b92b9cd457ece139962a3cddaf99f60e167ced8c8107a1ea53983fd6a57: Status 404 returned error can't find the container with id c7b80b92b9cd457ece139962a3cddaf99f60e167ced8c8107a1ea53983fd6a57
Sep 16 21:51:36.225207 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:36.225174    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:51:36.225314 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:36.225239    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:51:36.225314 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:36.225265    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:51:36.927073 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:36.927033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f" event={"ID":"6978ea7c-6aae-4c24-adf9-6a1498e9aabc","Type":"ContainerStarted","Data":"5bbd5e31b1d0be45d3cf2869224e52c9f2a93a6f1262a7f6e06d030108b72a61"}
Sep 16 21:51:36.927464 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:36.927117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f" event={"ID":"6978ea7c-6aae-4c24-adf9-6a1498e9aabc","Type":"ContainerStarted","Data":"c7b80b92b9cd457ece139962a3cddaf99f60e167ced8c8107a1ea53983fd6a57"}
Sep 16 21:51:36.950441 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:36.950390    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f" podStartSLOduration=1.9503749350000001 podStartE2EDuration="1.950374935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:51:36.948939486 +0000 UTC m=+1756.849149489" watchObservedRunningTime="2026-09-16 21:51:36.950374935 +0000 UTC m=+1756.850584991"
Sep 16 21:51:37.921460 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:37.921420    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:37.926482 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:37.926455    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:37.931870 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:37.931852    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:37.932510 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:37.932493    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:42.961351 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:42.961316    2563 generic.go:356] "Generic (PLEG): container finished" podID="57903415-58fe-452e-ae2a-eb30704bcff0" containerID="5a9b340cb02cb4f2a0d2a95f3cbeb47ee4cd22be52adeaec0b82a7e4f602e559" exitCode=0
Sep 16 21:51:42.961820 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:42.961386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8" event={"ID":"57903415-58fe-452e-ae2a-eb30704bcff0","Type":"ContainerDied","Data":"5a9b340cb02cb4f2a0d2a95f3cbeb47ee4cd22be52adeaec0b82a7e4f602e559"}
Sep 16 21:51:42.961932 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:42.961915    2563 scope.go:117] "RemoveContainer" containerID="5a9b340cb02cb4f2a0d2a95f3cbeb47ee4cd22be52adeaec0b82a7e4f602e559"
Sep 16 21:51:42.963120 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:42.963090    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:51:42.963193 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:42.963160    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:51:42.963193 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:42.963189    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:51:43.971367 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:43.971313    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8" event={"ID":"57903415-58fe-452e-ae2a-eb30704bcff0","Type":"ContainerStarted","Data":"89ff965eca2b4037ffde0599604c16ac53fd02ed1c6f9f372500117bde21356f"}
Sep 16 21:51:44.016527 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:44.016490    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:44.016741 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:44.016719    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 16 21:51:44.016800 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:44.016779    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8" podUID="57903415-58fe-452e-ae2a-eb30704bcff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 16 21:51:45.016814 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:45.016777    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 16 21:51:45.017228 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:45.016842    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8" podUID="57903415-58fe-452e-ae2a-eb30704bcff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 16 21:51:46.016701 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:46.016641    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 16 21:51:46.016880 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:46.016721    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8" podUID="57903415-58fe-452e-ae2a-eb30704bcff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 16 21:51:47.016825 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:47.016790    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 16 21:51:47.016997 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:47.016852    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8" podUID="57903415-58fe-452e-ae2a-eb30704bcff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 16 21:51:48.016880 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:48.016830    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 16 21:51:48.017102 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:48.016905    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8" podUID="57903415-58fe-452e-ae2a-eb30704bcff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 16 21:51:49.017225 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:49.017179    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Sep 16 21:51:49.017620 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:49.017253    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8" podUID="57903415-58fe-452e-ae2a-eb30704bcff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Sep 16 21:51:50.020644 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:50.020615    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:50.021059 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:50.021042    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:50.021627 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:50.021601    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:51:57.051124 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:57.051087    2563 generic.go:356] "Generic (PLEG): container finished" podID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" containerID="5bbd5e31b1d0be45d3cf2869224e52c9f2a93a6f1262a7f6e06d030108b72a61" exitCode=0
Sep 16 21:51:57.051540 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:57.051162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f" event={"ID":"6978ea7c-6aae-4c24-adf9-6a1498e9aabc","Type":"ContainerDied","Data":"5bbd5e31b1d0be45d3cf2869224e52c9f2a93a6f1262a7f6e06d030108b72a61"}
Sep 16 21:51:57.051592 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:57.051577    2563 scope.go:117] "RemoveContainer" containerID="5bbd5e31b1d0be45d3cf2869224e52c9f2a93a6f1262a7f6e06d030108b72a61"
Sep 16 21:51:57.052715 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:57.052687    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:51:57.052774 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:57.052760    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:51:57.052811 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:57.052795    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:51:58.058774 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:58.058739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f" event={"ID":"6978ea7c-6aae-4c24-adf9-6a1498e9aabc","Type":"ContainerStarted","Data":"7cbdc71a6e9922502abfb6a7e482b8432e5f94b1e77ddae3bf4883550f3f322c"}
Sep 16 21:51:58.921344 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:58.921303    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:51:58.921568 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:58.921543    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 16 21:51:58.921620 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:58.921598    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f" podUID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 16 21:51:59.921014 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:59.920972    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 16 21:51:59.921492 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:51:59.921034    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f" podUID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 16 21:52:00.921639 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:00.921602    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 16 21:52:00.922062 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:00.921688    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f" podUID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 16 21:52:01.921849 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:01.921816    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 16 21:52:01.922232 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:01.921883    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f" podUID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 16 21:52:02.921404 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:02.921358    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 16 21:52:02.921602 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:02.921424    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f" podUID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 16 21:52:03.924979 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:03.924949    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:52:03.925391 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:03.925239    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:52:03.926046 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:03.926007    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:52:21.017594 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:21.017561    2563 scope.go:117] "RemoveContainer" containerID="a3c102ab31390a1b58aa6302ea9dbf5eb059ec90ea6751cc1e2f2f51da9193ae"
Sep 16 21:52:21.025888 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:21.025870    2563 scope.go:117] "RemoveContainer" containerID="742c84b5e5adf02c57171325512df79b1c7c3b17ba2777de11f209aabc985914"
Sep 16 21:52:22.178059 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.178025    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"]
Sep 16 21:52:22.182965 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.182936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.185969 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.185947    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ad0222\"/\"default-dockercfg-8jspj\""
Sep 16 21:52:22.186121 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.185983    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ad0222\"/\"openshift-service-ca.crt\""
Sep 16 21:52:22.186121 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.186007    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ad0222\"/\"kube-root-ca.crt\""
Sep 16 21:52:22.186121 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.186026    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ad0222\"/\"ipp-route-ad0222-kserve-self-signed-certs\""
Sep 16 21:52:22.190986 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.190964    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"]
Sep 16 21:52:22.259252 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.259219    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c6ddac-dcc4-4516-a812-07a9c885846f-tls-certs\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.259252 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.259254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-tmp-dir\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.259460 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.259280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-home\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.259460 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.259410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9qgx\" (UniqueName: \"kubernetes.io/projected/d2c6ddac-dcc4-4516-a812-07a9c885846f-kube-api-access-f9qgx\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.259460 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.259448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-model-cache\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.259563 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.259502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-dshm\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.360679 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.360622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9qgx\" (UniqueName: \"kubernetes.io/projected/d2c6ddac-dcc4-4516-a812-07a9c885846f-kube-api-access-f9qgx\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.360859 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.360697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-model-cache\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.360859 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.360722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-dshm\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.360859 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.360761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c6ddac-dcc4-4516-a812-07a9c885846f-tls-certs\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.360859 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.360789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-tmp-dir\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.360859 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.360822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-home\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.361144 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.361126    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-model-cache\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.361204 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.361183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-tmp-dir\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.361289 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.361274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-home\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.363209 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.363193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-dshm\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.363305 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.363291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c6ddac-dcc4-4516-a812-07a9c885846f-tls-certs\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.368145 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.368125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9qgx\" (UniqueName: \"kubernetes.io/projected/d2c6ddac-dcc4-4516-a812-07a9c885846f-kube-api-access-f9qgx\") pod \"ipp-route-ad0222-kserve-b545d64dc-twwtx\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") " pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.493592 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.493486    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:22.766428 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:22.766400    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"]
Sep 16 21:52:22.768346 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:52:22.768320    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2c6ddac_dcc4_4516_a812_07a9c885846f.slice/crio-b56fa7286125e5ee34c9c8a2b1312e68f593db09a31794c5f3fc35170d19b601 WatchSource:0}: Error finding container b56fa7286125e5ee34c9c8a2b1312e68f593db09a31794c5f3fc35170d19b601: Status 404 returned error can't find the container with id b56fa7286125e5ee34c9c8a2b1312e68f593db09a31794c5f3fc35170d19b601
Sep 16 21:52:23.215105 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:23.215069    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx" event={"ID":"d2c6ddac-dcc4-4516-a812-07a9c885846f","Type":"ContainerStarted","Data":"9b3e344cd92b2f6be2e5c0de1f5af1a5441616fbd32d26d3ee4f53a8598d7ca5"}
Sep 16 21:52:23.215105 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:23.215117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx" event={"ID":"d2c6ddac-dcc4-4516-a812-07a9c885846f","Type":"ContainerStarted","Data":"b56fa7286125e5ee34c9c8a2b1312e68f593db09a31794c5f3fc35170d19b601"}
Sep 16 21:52:23.232422 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:23.232372    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx" podStartSLOduration=1.232357207 podStartE2EDuration="1.232357207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:52:23.231439801 +0000 UTC m=+1803.131649807" watchObservedRunningTime="2026-09-16 21:52:23.232357207 +0000 UTC m=+1803.132567211"
Sep 16 21:52:32.494577 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:32.494540    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:32.495151 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:32.494594    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:32.507339 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:32.507311    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:33.272384 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:33.272357    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:52:45.751916 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.751882    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"]
Sep 16 21:52:45.755631 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.755610    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.757914 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.757892    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-dcbfbc-openshift-default-dockercfg-dk5x8\""
Sep 16 21:52:45.766742 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.766719    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"]
Sep 16 21:52:45.792322 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.792283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c656493-fbbe-4f82-82c2-c56a0319e728-istiod-ca-cert\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.792479 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.792330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq8dp\" (UniqueName: \"kubernetes.io/projected/2c656493-fbbe-4f82-82c2-c56a0319e728-kube-api-access-fq8dp\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.792479 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.792395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-envoy\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.792479 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.792419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-credential-socket\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.792479 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.792470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-data\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.792637 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.792494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-workload-socket\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.792637 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.792519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-workload-certs\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.792637 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.792553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-token\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.792637 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.792567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-podinfo\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.893485 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.893441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-podinfo\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.893721 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.893506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c656493-fbbe-4f82-82c2-c56a0319e728-istiod-ca-cert\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.893721 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.893540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq8dp\" (UniqueName: \"kubernetes.io/projected/2c656493-fbbe-4f82-82c2-c56a0319e728-kube-api-access-fq8dp\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.893721 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.893599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-envoy\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.893721 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.893634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-credential-socket\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.893971 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.893727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-data\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.893971 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.893764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-workload-socket\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.893971 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.893802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-workload-certs\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.893971 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.893864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-token\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.894173 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.894043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-credential-socket\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.894261 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.894238    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-workload-socket\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.894298 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.894262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-data\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.894343 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.894301    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-workload-certs\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.894382 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.894345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c656493-fbbe-4f82-82c2-c56a0319e728-istiod-ca-cert\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.896021 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.896003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-podinfo\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.896118 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.896097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-envoy\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.902447 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.902420    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq8dp\" (UniqueName: \"kubernetes.io/projected/2c656493-fbbe-4f82-82c2-c56a0319e728-kube-api-access-fq8dp\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:45.902588 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:45.902570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-token\") pod \"e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") " pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:46.067046 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:46.066951    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:46.349115 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:46.349086    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"]
Sep 16 21:52:46.350518 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:52:46.350488    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c656493_fbbe_4f82_82c2_c56a0319e728.slice/crio-4c885643d2bdc473f019c94a9b139b0bca07ba594aa852d1131bdac5ca8b601c WatchSource:0}: Error finding container 4c885643d2bdc473f019c94a9b139b0bca07ba594aa852d1131bdac5ca8b601c: Status 404 returned error can't find the container with id 4c885643d2bdc473f019c94a9b139b0bca07ba594aa852d1131bdac5ca8b601c
Sep 16 21:52:46.352922 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:46.352888    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:52:46.353005 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:46.352956    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:52:46.353005 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:46.352983    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:52:47.334797 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:47.334765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf" event={"ID":"2c656493-fbbe-4f82-82c2-c56a0319e728","Type":"ContainerStarted","Data":"4de400fe3a87f729f0fa65fcfc6275d0a0d563816870d212a323bf12f56d3803"}
Sep 16 21:52:47.335194 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:47.334821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf" event={"ID":"2c656493-fbbe-4f82-82c2-c56a0319e728","Type":"ContainerStarted","Data":"4c885643d2bdc473f019c94a9b139b0bca07ba594aa852d1131bdac5ca8b601c"}
Sep 16 21:52:47.359729 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:47.359674    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf" podStartSLOduration=2.359646616 podStartE2EDuration="2.359646616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:52:47.357130468 +0000 UTC m=+1827.257340474" watchObservedRunningTime="2026-09-16 21:52:47.359646616 +0000 UTC m=+1827.259856619"
Sep 16 21:52:48.067574 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:48.067533    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:48.072792 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:48.072764    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:48.340857 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:48.340778    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:52:48.341736 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:52:48.341718    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:53:09.454260 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:09.454226    2563 generic.go:356] "Generic (PLEG): container finished" podID="2c656493-fbbe-4f82-82c2-c56a0319e728" containerID="4de400fe3a87f729f0fa65fcfc6275d0a0d563816870d212a323bf12f56d3803" exitCode=0
Sep 16 21:53:09.454817 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:09.454292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf" event={"ID":"2c656493-fbbe-4f82-82c2-c56a0319e728","Type":"ContainerDied","Data":"4de400fe3a87f729f0fa65fcfc6275d0a0d563816870d212a323bf12f56d3803"}
Sep 16 21:53:09.454886 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:09.454870    2563 scope.go:117] "RemoveContainer" containerID="4de400fe3a87f729f0fa65fcfc6275d0a0d563816870d212a323bf12f56d3803"
Sep 16 21:53:09.456063 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:09.456021    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:53:09.456179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:09.456107    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:53:09.456179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:09.456157    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 21:53:10.461312 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:10.461279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf" event={"ID":"2c656493-fbbe-4f82-82c2-c56a0319e728","Type":"ContainerStarted","Data":"1610e947a8a371e5c174c6f2013cdd9b8cdeb20373fd7cd6556f675960283eee"}
Sep 16 21:53:11.067622 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:11.067581    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:53:11.067863 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:11.067740    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 16 21:53:11.067863 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:11.067788    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf" podUID="2c656493-fbbe-4f82-82c2-c56a0319e728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 16 21:53:12.067961 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:12.067926    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 16 21:53:12.068543 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:12.068512    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf" podUID="2c656493-fbbe-4f82-82c2-c56a0319e728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 16 21:53:13.067760 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:13.067722    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 16 21:53:13.068061 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:13.068030    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf" podUID="2c656493-fbbe-4f82-82c2-c56a0319e728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 16 21:53:14.068434 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:14.068389    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 16 21:53:14.068987 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:14.068461    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf" podUID="2c656493-fbbe-4f82-82c2-c56a0319e728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 16 21:53:15.067856 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:15.067814    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Sep 16 21:53:15.068196 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:15.068159    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf" podUID="2c656493-fbbe-4f82-82c2-c56a0319e728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Sep 16 21:53:16.078513 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:16.078475    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:53:16.079029 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:16.078875    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:53:16.080076 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:16.080057    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:53:20.014123 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:20.014083    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn"]
Sep 16 21:53:20.017782 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:20.017766    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn"
Sep 16 21:53:20.020021 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:20.019999    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:53:20.020154 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:20.020002    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:53:20.026145 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:20.025678    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn"]
Sep 16 21:53:20.060979 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:20.060943    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468d61e-c39f-4d46-970e-db1e2383ce68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn\" (UID: \"6468d61e-c39f-4d46-970e-db1e2383ce68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn"
Sep 16 21:53:20.161773 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:20.161734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468d61e-c39f-4d46-970e-db1e2383ce68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn\" (UID: \"6468d61e-c39f-4d46-970e-db1e2383ce68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn"
Sep 16 21:53:20.162407 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:20.162385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468d61e-c39f-4d46-970e-db1e2383ce68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn\" (UID: \"6468d61e-c39f-4d46-970e-db1e2383ce68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn"
Sep 16 21:53:20.329614 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:20.329529    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn"
Sep 16 21:53:20.599211 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:20.599185    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn"]
Sep 16 21:53:20.601266 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:53:20.601233    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6468d61e_c39f_4d46_970e_db1e2383ce68.slice/crio-8411275c312c2a9475e19401c82e967f7cc52e241524f55bc30ec5c6e0aa13ee WatchSource:0}: Error finding container 8411275c312c2a9475e19401c82e967f7cc52e241524f55bc30ec5c6e0aa13ee: Status 404 returned error can't find the container with id 8411275c312c2a9475e19401c82e967f7cc52e241524f55bc30ec5c6e0aa13ee
Sep 16 21:53:21.051880 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:21.051851    2563 scope.go:117] "RemoveContainer" containerID="63bf86fa942994d639253215f54fb1d762755774921fa13bb466313629d11ddb"
Sep 16 21:53:21.062711 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:21.062638    2563 scope.go:117] "RemoveContainer" containerID="627c48812a62f6ef7466f6a1253eea13af8c1c9f15ede0720dfbabf16cb483f3"
Sep 16 21:53:21.071821 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:21.071791    2563 scope.go:117] "RemoveContainer" containerID="f7194236b378809fa5ae65e8f0fb60f9117c501734cd66b9c9f2c5f02fad087b"
Sep 16 21:53:21.079436 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:21.079414    2563 scope.go:117] "RemoveContainer" containerID="9ca6975a39d0e7b1071022a83647badb6e8310efa57e7c35b74e9a459d494246"
Sep 16 21:53:21.530599 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:21.530563    2563 generic.go:356] "Generic (PLEG): container finished" podID="6468d61e-c39f-4d46-970e-db1e2383ce68" containerID="2ffc835c81f34a0a4e347fc8dd5c0f0c748f347ee7896617c222fdd47da261be" exitCode=0
Sep 16 21:53:21.530801 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:21.530628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn" event={"ID":"6468d61e-c39f-4d46-970e-db1e2383ce68","Type":"ContainerDied","Data":"2ffc835c81f34a0a4e347fc8dd5c0f0c748f347ee7896617c222fdd47da261be"}
Sep 16 21:53:21.530801 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:21.530708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn" event={"ID":"6468d61e-c39f-4d46-970e-db1e2383ce68","Type":"ContainerStarted","Data":"8411275c312c2a9475e19401c82e967f7cc52e241524f55bc30ec5c6e0aa13ee"}
Sep 16 21:53:22.665803 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:22.665773    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn"
Sep 16 21:53:22.792422 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:22.792321    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468d61e-c39f-4d46-970e-db1e2383ce68-maas-api-service-ca\") pod \"6468d61e-c39f-4d46-970e-db1e2383ce68\" (UID: \"6468d61e-c39f-4d46-970e-db1e2383ce68\") "
Sep 16 21:53:22.792746 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:22.792720    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6468d61e-c39f-4d46-970e-db1e2383ce68-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6468d61e-c39f-4d46-970e-db1e2383ce68" (UID: "6468d61e-c39f-4d46-970e-db1e2383ce68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:53:22.893933 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:22.893891    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468d61e-c39f-4d46-970e-db1e2383ce68-maas-api-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:53:23.541140 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:23.541112    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn"
Sep 16 21:53:23.541321 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:23.541112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn" event={"ID":"6468d61e-c39f-4d46-970e-db1e2383ce68","Type":"ContainerDied","Data":"8411275c312c2a9475e19401c82e967f7cc52e241524f55bc30ec5c6e0aa13ee"}
Sep 16 21:53:23.541321 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:23.541273    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8411275c312c2a9475e19401c82e967f7cc52e241524f55bc30ec5c6e0aa13ee"
Sep 16 21:53:31.070061 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:31.070023    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn"]
Sep 16 21:53:31.072930 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:31.072901    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-dcbfbc-c4bdn"]
Sep 16 21:53:32.608772 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:32.608734    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6468d61e-c39f-4d46-970e-db1e2383ce68" path="/var/lib/kubelet/pods/6468d61e-c39f-4d46-970e-db1e2383ce68/volumes"
Sep 16 21:53:37.637081 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:37.637052    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"]
Sep 16 21:53:37.637481 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:37.637302    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf" podUID="2c656493-fbbe-4f82-82c2-c56a0319e728" containerName="istio-proxy" containerID="cri-o://1610e947a8a371e5c174c6f2013cdd9b8cdeb20373fd7cd6556f675960283eee" gracePeriod=30
Sep 16 21:53:41.135602 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:41.135569    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4"]
Sep 16 21:53:41.135996 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:41.135972    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6468d61e-c39f-4d46-970e-db1e2383ce68" containerName="revoke-keys"
Sep 16 21:53:41.136045 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:41.135997    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6468d61e-c39f-4d46-970e-db1e2383ce68" containerName="revoke-keys"
Sep 16 21:53:41.136085 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:41.136068    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6468d61e-c39f-4d46-970e-db1e2383ce68" containerName="revoke-keys"
Sep 16 21:53:41.138932 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:41.138916    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4"
Sep 16 21:53:41.141099 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:41.141080    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:53:41.141216 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:41.141083    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:53:41.146888 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:41.146863    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4"]
Sep 16 21:53:41.185907 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:41.185867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b79805-8f96-4a24-bd6e-595bd5fcfa5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4\" (UID: \"50b79805-8f96-4a24-bd6e-595bd5fcfa5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4"
Sep 16 21:53:41.287450 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:41.287413    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b79805-8f96-4a24-bd6e-595bd5fcfa5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4\" (UID: \"50b79805-8f96-4a24-bd6e-595bd5fcfa5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4"
Sep 16 21:53:41.288125 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:41.288098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b79805-8f96-4a24-bd6e-595bd5fcfa5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4\" (UID: \"50b79805-8f96-4a24-bd6e-595bd5fcfa5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4"
Sep 16 21:53:41.450321 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:41.450282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4"
Sep 16 21:53:41.721825 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:41.721800    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4"]
Sep 16 21:53:42.637262 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:42.637178    2563 generic.go:356] "Generic (PLEG): container finished" podID="50b79805-8f96-4a24-bd6e-595bd5fcfa5d" containerID="99e16b0ed0e100fd89134d2d493b3aa594c3ae3eccdc5b2104e32ee943d8068e" exitCode=0
Sep 16 21:53:42.637262 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:42.637224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4" event={"ID":"50b79805-8f96-4a24-bd6e-595bd5fcfa5d","Type":"ContainerDied","Data":"99e16b0ed0e100fd89134d2d493b3aa594c3ae3eccdc5b2104e32ee943d8068e"}
Sep 16 21:53:42.637701 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:42.637272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4" event={"ID":"50b79805-8f96-4a24-bd6e-595bd5fcfa5d","Type":"ContainerStarted","Data":"91d3db0fcea86233dd198f23ccaa27e410da490bd3fe0df7c7c579742230d2d5"}
Sep 16 21:53:42.901564 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:42.901498    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:53:43.003391 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.003344    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-token\") pod \"2c656493-fbbe-4f82-82c2-c56a0319e728\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") "
Sep 16 21:53:43.003588 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.003420    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-podinfo\") pod \"2c656493-fbbe-4f82-82c2-c56a0319e728\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") "
Sep 16 21:53:43.003588 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.003444    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c656493-fbbe-4f82-82c2-c56a0319e728-istiod-ca-cert\") pod \"2c656493-fbbe-4f82-82c2-c56a0319e728\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") "
Sep 16 21:53:43.003588 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.003472    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-credential-socket\") pod \"2c656493-fbbe-4f82-82c2-c56a0319e728\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") "
Sep 16 21:53:43.003588 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.003491    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-envoy\") pod \"2c656493-fbbe-4f82-82c2-c56a0319e728\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") "
Sep 16 21:53:43.003588 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.003517    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-workload-certs\") pod \"2c656493-fbbe-4f82-82c2-c56a0319e728\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") "
Sep 16 21:53:43.003588 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.003549    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-data\") pod \"2c656493-fbbe-4f82-82c2-c56a0319e728\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") "
Sep 16 21:53:43.003588 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.003567    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-workload-socket\") pod \"2c656493-fbbe-4f82-82c2-c56a0319e728\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") "
Sep 16 21:53:43.004013 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.003616    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq8dp\" (UniqueName: \"kubernetes.io/projected/2c656493-fbbe-4f82-82c2-c56a0319e728-kube-api-access-fq8dp\") pod \"2c656493-fbbe-4f82-82c2-c56a0319e728\" (UID: \"2c656493-fbbe-4f82-82c2-c56a0319e728\") "
Sep 16 21:53:43.004013 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.003872    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2c656493-fbbe-4f82-82c2-c56a0319e728" (UID: "2c656493-fbbe-4f82-82c2-c56a0319e728"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:53:43.004013 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.003968    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2c656493-fbbe-4f82-82c2-c56a0319e728" (UID: "2c656493-fbbe-4f82-82c2-c56a0319e728"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:53:43.004013 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.003985    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c656493-fbbe-4f82-82c2-c56a0319e728-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2c656493-fbbe-4f82-82c2-c56a0319e728" (UID: "2c656493-fbbe-4f82-82c2-c56a0319e728"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:53:43.004226 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.004025    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:53:43.004226 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.004045    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2c656493-fbbe-4f82-82c2-c56a0319e728" (UID: "2c656493-fbbe-4f82-82c2-c56a0319e728"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:53:43.004226 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.004059    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-data" (OuterVolumeSpecName: "istio-data") pod "2c656493-fbbe-4f82-82c2-c56a0319e728" (UID: "2c656493-fbbe-4f82-82c2-c56a0319e728"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:53:43.005833 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.005805    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2c656493-fbbe-4f82-82c2-c56a0319e728" (UID: "2c656493-fbbe-4f82-82c2-c56a0319e728"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:53:43.005949 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.005874    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2c656493-fbbe-4f82-82c2-c56a0319e728" (UID: "2c656493-fbbe-4f82-82c2-c56a0319e728"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:53:43.005949 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.005908    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-token" (OuterVolumeSpecName: "istio-token") pod "2c656493-fbbe-4f82-82c2-c56a0319e728" (UID: "2c656493-fbbe-4f82-82c2-c56a0319e728"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:53:43.006030 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.005971    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c656493-fbbe-4f82-82c2-c56a0319e728-kube-api-access-fq8dp" (OuterVolumeSpecName: "kube-api-access-fq8dp") pod "2c656493-fbbe-4f82-82c2-c56a0319e728" (UID: "2c656493-fbbe-4f82-82c2-c56a0319e728"). InnerVolumeSpecName "kube-api-access-fq8dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:53:43.105308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.105269    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq8dp\" (UniqueName: \"kubernetes.io/projected/2c656493-fbbe-4f82-82c2-c56a0319e728-kube-api-access-fq8dp\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:53:43.105308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.105299    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:53:43.105308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.105308    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:53:43.105308 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.105316    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c656493-fbbe-4f82-82c2-c56a0319e728-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:53:43.105580 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.105325    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:53:43.105580 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.105333    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:53:43.105580 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.105341    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:53:43.105580 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.105349    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c656493-fbbe-4f82-82c2-c56a0319e728-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:53:43.644802 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.644718    2563 generic.go:356] "Generic (PLEG): container finished" podID="2c656493-fbbe-4f82-82c2-c56a0319e728" containerID="1610e947a8a371e5c174c6f2013cdd9b8cdeb20373fd7cd6556f675960283eee" exitCode=0
Sep 16 21:53:43.644802 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.644764    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"
Sep 16 21:53:43.645277 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.644793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf" event={"ID":"2c656493-fbbe-4f82-82c2-c56a0319e728","Type":"ContainerDied","Data":"1610e947a8a371e5c174c6f2013cdd9b8cdeb20373fd7cd6556f675960283eee"}
Sep 16 21:53:43.645277 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.644853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf" event={"ID":"2c656493-fbbe-4f82-82c2-c56a0319e728","Type":"ContainerDied","Data":"4c885643d2bdc473f019c94a9b139b0bca07ba594aa852d1131bdac5ca8b601c"}
Sep 16 21:53:43.645277 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.644875    2563 scope.go:117] "RemoveContainer" containerID="1610e947a8a371e5c174c6f2013cdd9b8cdeb20373fd7cd6556f675960283eee"
Sep 16 21:53:43.654448 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.654429    2563 scope.go:117] "RemoveContainer" containerID="4de400fe3a87f729f0fa65fcfc6275d0a0d563816870d212a323bf12f56d3803"
Sep 16 21:53:43.663597 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.663499    2563 scope.go:117] "RemoveContainer" containerID="1610e947a8a371e5c174c6f2013cdd9b8cdeb20373fd7cd6556f675960283eee"
Sep 16 21:53:43.664177 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:53:43.664095    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1610e947a8a371e5c174c6f2013cdd9b8cdeb20373fd7cd6556f675960283eee\": container with ID starting with 1610e947a8a371e5c174c6f2013cdd9b8cdeb20373fd7cd6556f675960283eee not found: ID does not exist" containerID="1610e947a8a371e5c174c6f2013cdd9b8cdeb20373fd7cd6556f675960283eee"
Sep 16 21:53:43.664177 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.664137    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1610e947a8a371e5c174c6f2013cdd9b8cdeb20373fd7cd6556f675960283eee"} err="failed to get container status \"1610e947a8a371e5c174c6f2013cdd9b8cdeb20373fd7cd6556f675960283eee\": rpc error: code = NotFound desc = could not find container \"1610e947a8a371e5c174c6f2013cdd9b8cdeb20373fd7cd6556f675960283eee\": container with ID starting with 1610e947a8a371e5c174c6f2013cdd9b8cdeb20373fd7cd6556f675960283eee not found: ID does not exist"
Sep 16 21:53:43.664177 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.664162    2563 scope.go:117] "RemoveContainer" containerID="4de400fe3a87f729f0fa65fcfc6275d0a0d563816870d212a323bf12f56d3803"
Sep 16 21:53:43.664463 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:53:43.664444    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4de400fe3a87f729f0fa65fcfc6275d0a0d563816870d212a323bf12f56d3803\": container with ID starting with 4de400fe3a87f729f0fa65fcfc6275d0a0d563816870d212a323bf12f56d3803 not found: ID does not exist" containerID="4de400fe3a87f729f0fa65fcfc6275d0a0d563816870d212a323bf12f56d3803"
Sep 16 21:53:43.664527 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.664467    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4de400fe3a87f729f0fa65fcfc6275d0a0d563816870d212a323bf12f56d3803"} err="failed to get container status \"4de400fe3a87f729f0fa65fcfc6275d0a0d563816870d212a323bf12f56d3803\": rpc error: code = NotFound desc = could not find container \"4de400fe3a87f729f0fa65fcfc6275d0a0d563816870d212a323bf12f56d3803\": container with ID starting with 4de400fe3a87f729f0fa65fcfc6275d0a0d563816870d212a323bf12f56d3803 not found: ID does not exist"
Sep 16 21:53:43.666190 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.666167    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"]
Sep 16 21:53:43.669844 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.669822    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-dcbfbc-openshift-default-7f8c88fb7b-469gf"]
Sep 16 21:53:43.781402 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.781369    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4"
Sep 16 21:53:43.813698 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.813640    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b79805-8f96-4a24-bd6e-595bd5fcfa5d-maas-api-service-ca\") pod \"50b79805-8f96-4a24-bd6e-595bd5fcfa5d\" (UID: \"50b79805-8f96-4a24-bd6e-595bd5fcfa5d\") "
Sep 16 21:53:43.814062 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.814034    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b79805-8f96-4a24-bd6e-595bd5fcfa5d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50b79805-8f96-4a24-bd6e-595bd5fcfa5d" (UID: "50b79805-8f96-4a24-bd6e-595bd5fcfa5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:53:43.915383 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:43.915273    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b79805-8f96-4a24-bd6e-595bd5fcfa5d-maas-api-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:53:44.608966 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:44.608936    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c656493-fbbe-4f82-82c2-c56a0319e728" path="/var/lib/kubelet/pods/2c656493-fbbe-4f82-82c2-c56a0319e728/volumes"
Sep 16 21:53:44.650949 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:44.650923    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4"
Sep 16 21:53:44.651391 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:44.650965    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4" event={"ID":"50b79805-8f96-4a24-bd6e-595bd5fcfa5d","Type":"ContainerDied","Data":"91d3db0fcea86233dd198f23ccaa27e410da490bd3fe0df7c7c579742230d2d5"}
Sep 16 21:53:44.651391 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:44.651015    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91d3db0fcea86233dd198f23ccaa27e410da490bd3fe0df7c7c579742230d2d5"
Sep 16 21:53:57.005674 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:57.005624    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4"]
Sep 16 21:53:57.012749 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:57.012725    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ad0222-jfnb4"]
Sep 16 21:53:58.608792 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:53:58.608757    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50b79805-8f96-4a24-bd6e-595bd5fcfa5d" path="/var/lib/kubelet/pods/50b79805-8f96-4a24-bd6e-595bd5fcfa5d/volumes"
Sep 16 21:54:02.339175 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:02.339137    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"]
Sep 16 21:54:17.358410 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.358356    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx" podUID="d2c6ddac-dcc4-4516-a812-07a9c885846f" containerName="main" containerID="cri-o://9b3e344cd92b2f6be2e5c0de1f5af1a5441616fbd32d26d3ee4f53a8598d7ca5" gracePeriod=45
Sep 16 21:54:17.613375 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.613305    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:54:17.658505 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.658468    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c6ddac-dcc4-4516-a812-07a9c885846f-tls-certs\") pod \"d2c6ddac-dcc4-4516-a812-07a9c885846f\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") "
Sep 16 21:54:17.658720 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.658525    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-dshm\") pod \"d2c6ddac-dcc4-4516-a812-07a9c885846f\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") "
Sep 16 21:54:17.658720 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.658612    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-tmp-dir\") pod \"d2c6ddac-dcc4-4516-a812-07a9c885846f\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") "
Sep 16 21:54:17.658720 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.658696    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9qgx\" (UniqueName: \"kubernetes.io/projected/d2c6ddac-dcc4-4516-a812-07a9c885846f-kube-api-access-f9qgx\") pod \"d2c6ddac-dcc4-4516-a812-07a9c885846f\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") "
Sep 16 21:54:17.658914 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.658731    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-home\") pod \"d2c6ddac-dcc4-4516-a812-07a9c885846f\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") "
Sep 16 21:54:17.658914 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.658773    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-model-cache\") pod \"d2c6ddac-dcc4-4516-a812-07a9c885846f\" (UID: \"d2c6ddac-dcc4-4516-a812-07a9c885846f\") "
Sep 16 21:54:17.659125 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.659007    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2c6ddac-dcc4-4516-a812-07a9c885846f" (UID: "d2c6ddac-dcc4-4516-a812-07a9c885846f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:54:17.659266 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.659166    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-home" (OuterVolumeSpecName: "home") pod "d2c6ddac-dcc4-4516-a812-07a9c885846f" (UID: "d2c6ddac-dcc4-4516-a812-07a9c885846f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:54:17.659266 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.659236    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2c6ddac-dcc4-4516-a812-07a9c885846f" (UID: "d2c6ddac-dcc4-4516-a812-07a9c885846f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:54:17.659547 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.659525    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-home\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:54:17.659624 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.659553    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-model-cache\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:54:17.659624 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.659569    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-tmp-dir\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:54:17.660989 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.660961    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c6ddac-dcc4-4516-a812-07a9c885846f-kube-api-access-f9qgx" (OuterVolumeSpecName: "kube-api-access-f9qgx") pod "d2c6ddac-dcc4-4516-a812-07a9c885846f" (UID: "d2c6ddac-dcc4-4516-a812-07a9c885846f"). InnerVolumeSpecName "kube-api-access-f9qgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:54:17.661088 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.661008    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-dshm" (OuterVolumeSpecName: "dshm") pod "d2c6ddac-dcc4-4516-a812-07a9c885846f" (UID: "d2c6ddac-dcc4-4516-a812-07a9c885846f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:54:17.661143 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.661100    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2c6ddac-dcc4-4516-a812-07a9c885846f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2c6ddac-dcc4-4516-a812-07a9c885846f" (UID: "d2c6ddac-dcc4-4516-a812-07a9c885846f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:54:17.760116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.760078    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9qgx\" (UniqueName: \"kubernetes.io/projected/d2c6ddac-dcc4-4516-a812-07a9c885846f-kube-api-access-f9qgx\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:54:17.760116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.760111    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c6ddac-dcc4-4516-a812-07a9c885846f-tls-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:54:17.760116 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.760121    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c6ddac-dcc4-4516-a812-07a9c885846f-dshm\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:54:17.817166 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.817128    2563 generic.go:356] "Generic (PLEG): container finished" podID="d2c6ddac-dcc4-4516-a812-07a9c885846f" containerID="9b3e344cd92b2f6be2e5c0de1f5af1a5441616fbd32d26d3ee4f53a8598d7ca5" exitCode=0
Sep 16 21:54:17.817328 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.817179    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx" event={"ID":"d2c6ddac-dcc4-4516-a812-07a9c885846f","Type":"ContainerDied","Data":"9b3e344cd92b2f6be2e5c0de1f5af1a5441616fbd32d26d3ee4f53a8598d7ca5"}
Sep 16 21:54:17.817328 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.817213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx" event={"ID":"d2c6ddac-dcc4-4516-a812-07a9c885846f","Type":"ContainerDied","Data":"b56fa7286125e5ee34c9c8a2b1312e68f593db09a31794c5f3fc35170d19b601"}
Sep 16 21:54:17.817328 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.817229    2563 scope.go:117] "RemoveContainer" containerID="9b3e344cd92b2f6be2e5c0de1f5af1a5441616fbd32d26d3ee4f53a8598d7ca5"
Sep 16 21:54:17.817328 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.817186    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"
Sep 16 21:54:17.826631 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.826609    2563 scope.go:117] "RemoveContainer" containerID="9b3e344cd92b2f6be2e5c0de1f5af1a5441616fbd32d26d3ee4f53a8598d7ca5"
Sep 16 21:54:17.826923 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:54:17.826905    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3e344cd92b2f6be2e5c0de1f5af1a5441616fbd32d26d3ee4f53a8598d7ca5\": container with ID starting with 9b3e344cd92b2f6be2e5c0de1f5af1a5441616fbd32d26d3ee4f53a8598d7ca5 not found: ID does not exist" containerID="9b3e344cd92b2f6be2e5c0de1f5af1a5441616fbd32d26d3ee4f53a8598d7ca5"
Sep 16 21:54:17.826977 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.826932    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3e344cd92b2f6be2e5c0de1f5af1a5441616fbd32d26d3ee4f53a8598d7ca5"} err="failed to get container status \"9b3e344cd92b2f6be2e5c0de1f5af1a5441616fbd32d26d3ee4f53a8598d7ca5\": rpc error: code = NotFound desc = could not find container \"9b3e344cd92b2f6be2e5c0de1f5af1a5441616fbd32d26d3ee4f53a8598d7ca5\": container with ID starting with 9b3e344cd92b2f6be2e5c0de1f5af1a5441616fbd32d26d3ee4f53a8598d7ca5 not found: ID does not exist"
Sep 16 21:54:17.838288 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.838259    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"]
Sep 16 21:54:17.844211 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:17.844185    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-ad0222/ipp-route-ad0222-kserve-b545d64dc-twwtx"]
Sep 16 21:54:18.608607 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:18.608571    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2c6ddac-dcc4-4516-a812-07a9c885846f" path="/var/lib/kubelet/pods/d2c6ddac-dcc4-4516-a812-07a9c885846f/volumes"
Sep 16 21:54:45.841875 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:45.841841    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"]
Sep 16 21:54:45.842399 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:45.842056    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8" podUID="57903415-58fe-452e-ae2a-eb30704bcff0" containerName="istio-proxy" containerID="cri-o://89ff965eca2b4037ffde0599604c16ac53fd02ed1c6f9f372500117bde21356f" gracePeriod=30
Sep 16 21:54:50.023508 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:50.023471    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 21:54:50.023906 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:50.023532    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8" podUID="57903415-58fe-452e-ae2a-eb30704bcff0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 21:54:50.983206 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:50.983172    2563 generic.go:356] "Generic (PLEG): container finished" podID="57903415-58fe-452e-ae2a-eb30704bcff0" containerID="89ff965eca2b4037ffde0599604c16ac53fd02ed1c6f9f372500117bde21356f" exitCode=0
Sep 16 21:54:50.983372 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:50.983240    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8" event={"ID":"57903415-58fe-452e-ae2a-eb30704bcff0","Type":"ContainerDied","Data":"89ff965eca2b4037ffde0599604c16ac53fd02ed1c6f9f372500117bde21356f"}
Sep 16 21:54:50.983372 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:50.983304    2563 scope.go:117] "RemoveContainer" containerID="5a9b340cb02cb4f2a0d2a95f3cbeb47ee4cd22be52adeaec0b82a7e4f602e559"
Sep 16 21:54:51.087356 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.087335    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:54:51.203713 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.203680    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57903415-58fe-452e-ae2a-eb30704bcff0-istio-podinfo\") pod \"57903415-58fe-452e-ae2a-eb30704bcff0\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") "
Sep 16 21:54:51.203904 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.203719    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-istio-envoy\") pod \"57903415-58fe-452e-ae2a-eb30704bcff0\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") "
Sep 16 21:54:51.203904 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.203739    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-istio-data\") pod \"57903415-58fe-452e-ae2a-eb30704bcff0\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") "
Sep 16 21:54:51.203904 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.203757    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-workload-socket\") pod \"57903415-58fe-452e-ae2a-eb30704bcff0\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") "
Sep 16 21:54:51.203904 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.203793    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-workload-certs\") pod \"57903415-58fe-452e-ae2a-eb30704bcff0\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") "
Sep 16 21:54:51.203904 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.203838    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57903415-58fe-452e-ae2a-eb30704bcff0-istiod-ca-cert\") pod \"57903415-58fe-452e-ae2a-eb30704bcff0\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") "
Sep 16 21:54:51.203904 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.203886    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57903415-58fe-452e-ae2a-eb30704bcff0-istio-token\") pod \"57903415-58fe-452e-ae2a-eb30704bcff0\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") "
Sep 16 21:54:51.204229 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.203933    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-credential-socket\") pod \"57903415-58fe-452e-ae2a-eb30704bcff0\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") "
Sep 16 21:54:51.204229 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.203982    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gcmv\" (UniqueName: \"kubernetes.io/projected/57903415-58fe-452e-ae2a-eb30704bcff0-kube-api-access-4gcmv\") pod \"57903415-58fe-452e-ae2a-eb30704bcff0\" (UID: \"57903415-58fe-452e-ae2a-eb30704bcff0\") "
Sep 16 21:54:51.204229 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.204055    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-istio-data" (OuterVolumeSpecName: "istio-data") pod "57903415-58fe-452e-ae2a-eb30704bcff0" (UID: "57903415-58fe-452e-ae2a-eb30704bcff0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:54:51.204229 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.204144    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "57903415-58fe-452e-ae2a-eb30704bcff0" (UID: "57903415-58fe-452e-ae2a-eb30704bcff0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:54:51.204229 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.204158    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "57903415-58fe-452e-ae2a-eb30704bcff0" (UID: "57903415-58fe-452e-ae2a-eb30704bcff0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:54:51.204503 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.204380    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:54:51.204503 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.204408    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:54:51.204503 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.204424    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:54:51.204503 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.204377    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "57903415-58fe-452e-ae2a-eb30704bcff0" (UID: "57903415-58fe-452e-ae2a-eb30704bcff0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:54:51.204503 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.204486    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57903415-58fe-452e-ae2a-eb30704bcff0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "57903415-58fe-452e-ae2a-eb30704bcff0" (UID: "57903415-58fe-452e-ae2a-eb30704bcff0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:54:51.206118 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.206096    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57903415-58fe-452e-ae2a-eb30704bcff0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "57903415-58fe-452e-ae2a-eb30704bcff0" (UID: "57903415-58fe-452e-ae2a-eb30704bcff0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:54:51.206118 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.206113    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57903415-58fe-452e-ae2a-eb30704bcff0-kube-api-access-4gcmv" (OuterVolumeSpecName: "kube-api-access-4gcmv") pod "57903415-58fe-452e-ae2a-eb30704bcff0" (UID: "57903415-58fe-452e-ae2a-eb30704bcff0"). InnerVolumeSpecName "kube-api-access-4gcmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:54:51.206351 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.206335    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "57903415-58fe-452e-ae2a-eb30704bcff0" (UID: "57903415-58fe-452e-ae2a-eb30704bcff0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:54:51.206501 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.206482    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57903415-58fe-452e-ae2a-eb30704bcff0-istio-token" (OuterVolumeSpecName: "istio-token") pod "57903415-58fe-452e-ae2a-eb30704bcff0" (UID: "57903415-58fe-452e-ae2a-eb30704bcff0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:54:51.305504 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.305462    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57903415-58fe-452e-ae2a-eb30704bcff0-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:54:51.305504 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.305498    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:54:51.305504 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.305512    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gcmv\" (UniqueName: \"kubernetes.io/projected/57903415-58fe-452e-ae2a-eb30704bcff0-kube-api-access-4gcmv\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:54:51.305805 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.305524    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57903415-58fe-452e-ae2a-eb30704bcff0-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:54:51.305805 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.305535    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57903415-58fe-452e-ae2a-eb30704bcff0-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:54:51.305805 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.305546    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57903415-58fe-452e-ae2a-eb30704bcff0-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:54:51.989186 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.989160    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"
Sep 16 21:54:51.989186 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.989172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8" event={"ID":"57903415-58fe-452e-ae2a-eb30704bcff0","Type":"ContainerDied","Data":"75277ce60a8351b4d5fc84da2830b7703173a84954d1cc636fe3d2b6bfb42449"}
Sep 16 21:54:51.989569 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:51.989223    2563 scope.go:117] "RemoveContainer" containerID="89ff965eca2b4037ffde0599604c16ac53fd02ed1c6f9f372500117bde21356f"
Sep 16 21:54:52.011515 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:52.011485    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"]
Sep 16 21:54:52.015856 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:52.015834    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ad0222-openshift-default-778686b7d4-h2px8"]
Sep 16 21:54:52.608520 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:54:52.608484    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57903415-58fe-452e-ae2a-eb30704bcff0" path="/var/lib/kubelet/pods/57903415-58fe-452e-ae2a-eb30704bcff0/volumes"
Sep 16 21:55:03.589003 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:03.588965    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"]
Sep 16 21:55:03.589448 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:03.589203    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f" podUID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" containerName="istio-proxy" containerID="cri-o://7cbdc71a6e9922502abfb6a7e482b8432e5f94b1e77ddae3bf4883550f3f322c" gracePeriod=30
Sep 16 21:55:03.927520 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:03.927484    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 21:55:03.927710 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:03.927544    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f" podUID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 21:55:08.831097 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.831071    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:55:08.985097 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.985057    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-podinfo\") pod \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") "
Sep 16 21:55:08.985322 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.985118    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-data\") pod \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") "
Sep 16 21:55:08.985322 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.985139    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istiod-ca-cert\") pod \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") "
Sep 16 21:55:08.985322 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.985171    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-token\") pod \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") "
Sep 16 21:55:08.985459 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.985376    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-envoy\") pod \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") "
Sep 16 21:55:08.985459 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.985428    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcjhh\" (UniqueName: \"kubernetes.io/projected/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-kube-api-access-mcjhh\") pod \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") "
Sep 16 21:55:08.985557 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.985478    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-credential-socket\") pod \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") "
Sep 16 21:55:08.985557 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.985374    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-data" (OuterVolumeSpecName: "istio-data") pod "6978ea7c-6aae-4c24-adf9-6a1498e9aabc" (UID: "6978ea7c-6aae-4c24-adf9-6a1498e9aabc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:55:08.985557 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.985517    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-workload-certs\") pod \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") "
Sep 16 21:55:08.985762 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.985589    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-workload-socket\") pod \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\" (UID: \"6978ea7c-6aae-4c24-adf9-6a1498e9aabc\") "
Sep 16 21:55:08.985762 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.985695    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6978ea7c-6aae-4c24-adf9-6a1498e9aabc" (UID: "6978ea7c-6aae-4c24-adf9-6a1498e9aabc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:55:08.985870 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.985776    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6978ea7c-6aae-4c24-adf9-6a1498e9aabc" (UID: "6978ea7c-6aae-4c24-adf9-6a1498e9aabc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:55:08.985931 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.985889    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6978ea7c-6aae-4c24-adf9-6a1498e9aabc" (UID: "6978ea7c-6aae-4c24-adf9-6a1498e9aabc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:55:08.985986 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.985946    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6978ea7c-6aae-4c24-adf9-6a1498e9aabc" (UID: "6978ea7c-6aae-4c24-adf9-6a1498e9aabc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:55:08.986122 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.986106    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:08.986173 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.986127    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:08.986173 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.986141    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:08.986173 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.986155    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:08.986173 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.986169    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:08.987642 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.987618    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6978ea7c-6aae-4c24-adf9-6a1498e9aabc" (UID: "6978ea7c-6aae-4c24-adf9-6a1498e9aabc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:55:08.987731 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.987678    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-kube-api-access-mcjhh" (OuterVolumeSpecName: "kube-api-access-mcjhh") pod "6978ea7c-6aae-4c24-adf9-6a1498e9aabc" (UID: "6978ea7c-6aae-4c24-adf9-6a1498e9aabc"). InnerVolumeSpecName "kube-api-access-mcjhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:55:08.987884 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.987867    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-token" (OuterVolumeSpecName: "istio-token") pod "6978ea7c-6aae-4c24-adf9-6a1498e9aabc" (UID: "6978ea7c-6aae-4c24-adf9-6a1498e9aabc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:55:08.987933 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:08.987880    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6978ea7c-6aae-4c24-adf9-6a1498e9aabc" (UID: "6978ea7c-6aae-4c24-adf9-6a1498e9aabc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:55:09.079104 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.079066    2563 generic.go:356] "Generic (PLEG): container finished" podID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" containerID="7cbdc71a6e9922502abfb6a7e482b8432e5f94b1e77ddae3bf4883550f3f322c" exitCode=0
Sep 16 21:55:09.079280 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.079137    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"
Sep 16 21:55:09.079280 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.079134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f" event={"ID":"6978ea7c-6aae-4c24-adf9-6a1498e9aabc","Type":"ContainerDied","Data":"7cbdc71a6e9922502abfb6a7e482b8432e5f94b1e77ddae3bf4883550f3f322c"}
Sep 16 21:55:09.079365 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.079295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f" event={"ID":"6978ea7c-6aae-4c24-adf9-6a1498e9aabc","Type":"ContainerDied","Data":"c7b80b92b9cd457ece139962a3cddaf99f60e167ced8c8107a1ea53983fd6a57"}
Sep 16 21:55:09.079365 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.079323    2563 scope.go:117] "RemoveContainer" containerID="7cbdc71a6e9922502abfb6a7e482b8432e5f94b1e77ddae3bf4883550f3f322c"
Sep 16 21:55:09.086938 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.086909    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:09.087087 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.086940    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:09.087087 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.086955    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcjhh\" (UniqueName: \"kubernetes.io/projected/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-kube-api-access-mcjhh\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:09.087087 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.086969    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6978ea7c-6aae-4c24-adf9-6a1498e9aabc-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:09.089014 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.088987    2563 scope.go:117] "RemoveContainer" containerID="5bbd5e31b1d0be45d3cf2869224e52c9f2a93a6f1262a7f6e06d030108b72a61"
Sep 16 21:55:09.097501 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.097476    2563 scope.go:117] "RemoveContainer" containerID="7cbdc71a6e9922502abfb6a7e482b8432e5f94b1e77ddae3bf4883550f3f322c"
Sep 16 21:55:09.097822 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:55:09.097803    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cbdc71a6e9922502abfb6a7e482b8432e5f94b1e77ddae3bf4883550f3f322c\": container with ID starting with 7cbdc71a6e9922502abfb6a7e482b8432e5f94b1e77ddae3bf4883550f3f322c not found: ID does not exist" containerID="7cbdc71a6e9922502abfb6a7e482b8432e5f94b1e77ddae3bf4883550f3f322c"
Sep 16 21:55:09.097895 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.097831    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cbdc71a6e9922502abfb6a7e482b8432e5f94b1e77ddae3bf4883550f3f322c"} err="failed to get container status \"7cbdc71a6e9922502abfb6a7e482b8432e5f94b1e77ddae3bf4883550f3f322c\": rpc error: code = NotFound desc = could not find container \"7cbdc71a6e9922502abfb6a7e482b8432e5f94b1e77ddae3bf4883550f3f322c\": container with ID starting with 7cbdc71a6e9922502abfb6a7e482b8432e5f94b1e77ddae3bf4883550f3f322c not found: ID does not exist"
Sep 16 21:55:09.097895 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.097851    2563 scope.go:117] "RemoveContainer" containerID="5bbd5e31b1d0be45d3cf2869224e52c9f2a93a6f1262a7f6e06d030108b72a61"
Sep 16 21:55:09.098131 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:55:09.098111    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bbd5e31b1d0be45d3cf2869224e52c9f2a93a6f1262a7f6e06d030108b72a61\": container with ID starting with 5bbd5e31b1d0be45d3cf2869224e52c9f2a93a6f1262a7f6e06d030108b72a61 not found: ID does not exist" containerID="5bbd5e31b1d0be45d3cf2869224e52c9f2a93a6f1262a7f6e06d030108b72a61"
Sep 16 21:55:09.098188 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.098141    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bbd5e31b1d0be45d3cf2869224e52c9f2a93a6f1262a7f6e06d030108b72a61"} err="failed to get container status \"5bbd5e31b1d0be45d3cf2869224e52c9f2a93a6f1262a7f6e06d030108b72a61\": rpc error: code = NotFound desc = could not find container \"5bbd5e31b1d0be45d3cf2869224e52c9f2a93a6f1262a7f6e06d030108b72a61\": container with ID starting with 5bbd5e31b1d0be45d3cf2869224e52c9f2a93a6f1262a7f6e06d030108b72a61 not found: ID does not exist"
Sep 16 21:55:09.102825 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.102791    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"]
Sep 16 21:55:09.107497 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:09.107472    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-03eca6-openshift-default-6969dcf7d8-28f4f"]
Sep 16 21:55:10.608824 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:10.608794    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" path="/var/lib/kubelet/pods/6978ea7c-6aae-4c24-adf9-6a1498e9aabc/volumes"
Sep 16 21:55:14.379223 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:14.379148    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7df81-29826585-2bk6d"]
Sep 16 21:55:14.381805 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:14.381776    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a7df81-29826585-2bk6d"]
Sep 16 21:55:14.607808 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:14.607772    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa28239f-af43-4194-b456-a662576fa0b7" path="/var/lib/kubelet/pods/fa28239f-af43-4194-b456-a662576fa0b7/volumes"
Sep 16 21:55:21.128685 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.128634    2563 scope.go:117] "RemoveContainer" containerID="67ddeead074ea1b8a74880941ec0d70eb302df0136d8a41d9ee381e3df7db14b"
Sep 16 21:55:21.224829 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.224792    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"]
Sep 16 21:55:21.225035 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.224996    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t" podUID="8fb9ef5a-b300-49de-8994-39e07ce01508" containerName="istio-proxy" containerID="cri-o://b7f3df3b4b2a19b4974a9536d4042196420017f1a51ac010902ed43ca8c7b23c" gracePeriod=30
Sep 16 21:55:21.532900 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.532863    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc"]
Sep 16 21:55:21.533212 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533199    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c656493-fbbe-4f82-82c2-c56a0319e728" containerName="istio-proxy"
Sep 16 21:55:21.533264 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533214    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c656493-fbbe-4f82-82c2-c56a0319e728" containerName="istio-proxy"
Sep 16 21:55:21.533264 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533230    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50b79805-8f96-4a24-bd6e-595bd5fcfa5d" containerName="revoke-keys"
Sep 16 21:55:21.533264 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533236    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50b79805-8f96-4a24-bd6e-595bd5fcfa5d" containerName="revoke-keys"
Sep 16 21:55:21.533264 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533244    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" containerName="istio-proxy"
Sep 16 21:55:21.533264 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533250    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" containerName="istio-proxy"
Sep 16 21:55:21.533264 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533259    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57903415-58fe-452e-ae2a-eb30704bcff0" containerName="istio-proxy"
Sep 16 21:55:21.533264 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533265    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="57903415-58fe-452e-ae2a-eb30704bcff0" containerName="istio-proxy"
Sep 16 21:55:21.533521 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533275    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c6ddac-dcc4-4516-a812-07a9c885846f" containerName="main"
Sep 16 21:55:21.533521 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533280    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c6ddac-dcc4-4516-a812-07a9c885846f" containerName="main"
Sep 16 21:55:21.533521 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533336    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c656493-fbbe-4f82-82c2-c56a0319e728" containerName="istio-proxy"
Sep 16 21:55:21.533521 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533342    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="50b79805-8f96-4a24-bd6e-595bd5fcfa5d" containerName="revoke-keys"
Sep 16 21:55:21.533521 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533350    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="57903415-58fe-452e-ae2a-eb30704bcff0" containerName="istio-proxy"
Sep 16 21:55:21.533521 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533355    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" containerName="istio-proxy"
Sep 16 21:55:21.533521 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533360    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="57903415-58fe-452e-ae2a-eb30704bcff0" containerName="istio-proxy"
Sep 16 21:55:21.533521 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533365    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c6ddac-dcc4-4516-a812-07a9c885846f" containerName="main"
Sep 16 21:55:21.533521 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.533370    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" containerName="istio-proxy"
Sep 16 21:55:21.538101 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.538077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc"
Sep 16 21:55:21.540213 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.540191    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:55:21.540366 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.540242    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 21:55:21.544788 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.544760    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc"]
Sep 16 21:55:21.603767 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.603731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83490087-7dfd-4fa8-9781-51ddc978cc20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc\" (UID: \"83490087-7dfd-4fa8-9781-51ddc978cc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc"
Sep 16 21:55:21.705018 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.704980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83490087-7dfd-4fa8-9781-51ddc978cc20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc\" (UID: \"83490087-7dfd-4fa8-9781-51ddc978cc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc"
Sep 16 21:55:21.706445 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.706411    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83490087-7dfd-4fa8-9781-51ddc978cc20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc\" (UID: \"83490087-7dfd-4fa8-9781-51ddc978cc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc"
Sep 16 21:55:21.851609 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:21.851512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc"
Sep 16 21:55:22.121072 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:22.120998    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc"]
Sep 16 21:55:22.123517 ip-10-0-136-118 kubenswrapper[2563]: W0916 21:55:22.123485    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83490087_7dfd_4fa8_9781_51ddc978cc20.slice/crio-7c060debe868b97973ae4e9d07339113ace535b13d14484d6933e046e3f3fa9d WatchSource:0}: Error finding container 7c060debe868b97973ae4e9d07339113ace535b13d14484d6933e046e3f3fa9d: Status 404 returned error can't find the container with id 7c060debe868b97973ae4e9d07339113ace535b13d14484d6933e046e3f3fa9d
Sep 16 21:55:22.146979 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:22.146950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc" event={"ID":"83490087-7dfd-4fa8-9781-51ddc978cc20","Type":"ContainerStarted","Data":"7c060debe868b97973ae4e9d07339113ace535b13d14484d6933e046e3f3fa9d"}
Sep 16 21:55:23.153051 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:23.152962    2563 generic.go:356] "Generic (PLEG): container finished" podID="83490087-7dfd-4fa8-9781-51ddc978cc20" containerID="3659c215d488ed1b57743220fd7910095e9263349aed94d98addb72eeed31bef" exitCode=0
Sep 16 21:55:23.153051 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:23.153002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc" event={"ID":"83490087-7dfd-4fa8-9781-51ddc978cc20","Type":"ContainerDied","Data":"3659c215d488ed1b57743220fd7910095e9263349aed94d98addb72eeed31bef"}
Sep 16 21:55:24.288196 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:24.288171    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc"
Sep 16 21:55:24.330709 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:24.330645    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83490087-7dfd-4fa8-9781-51ddc978cc20-maas-api-service-ca\") pod \"83490087-7dfd-4fa8-9781-51ddc978cc20\" (UID: \"83490087-7dfd-4fa8-9781-51ddc978cc20\") "
Sep 16 21:55:24.331027 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:24.330999    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83490087-7dfd-4fa8-9781-51ddc978cc20-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83490087-7dfd-4fa8-9781-51ddc978cc20" (UID: "83490087-7dfd-4fa8-9781-51ddc978cc20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:55:24.431882 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:24.431849    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83490087-7dfd-4fa8-9781-51ddc978cc20-maas-api-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:25.164548 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:25.164512    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc" event={"ID":"83490087-7dfd-4fa8-9781-51ddc978cc20","Type":"ContainerDied","Data":"7c060debe868b97973ae4e9d07339113ace535b13d14484d6933e046e3f3fa9d"}
Sep 16 21:55:25.164548 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:25.164535    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc"
Sep 16 21:55:25.164795 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:25.164588    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c060debe868b97973ae4e9d07339113ace535b13d14484d6933e046e3f3fa9d"
Sep 16 21:55:26.464208 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.464182    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:55:26.550777 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.550694    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-workload-certs\") pod \"8fb9ef5a-b300-49de-8994-39e07ce01508\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") "
Sep 16 21:55:26.550777 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.550735    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-workload-socket\") pod \"8fb9ef5a-b300-49de-8994-39e07ce01508\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") "
Sep 16 21:55:26.550777 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.550767    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-credential-socket\") pod \"8fb9ef5a-b300-49de-8994-39e07ce01508\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") "
Sep 16 21:55:26.551052 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.550793    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-data\") pod \"8fb9ef5a-b300-49de-8994-39e07ce01508\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") "
Sep 16 21:55:26.551052 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.550847    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-podinfo\") pod \"8fb9ef5a-b300-49de-8994-39e07ce01508\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") "
Sep 16 21:55:26.551052 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.550897    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mftbb\" (UniqueName: \"kubernetes.io/projected/8fb9ef5a-b300-49de-8994-39e07ce01508-kube-api-access-mftbb\") pod \"8fb9ef5a-b300-49de-8994-39e07ce01508\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") "
Sep 16 21:55:26.551052 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.550939    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-token\") pod \"8fb9ef5a-b300-49de-8994-39e07ce01508\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") "
Sep 16 21:55:26.551052 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.550981    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-envoy\") pod \"8fb9ef5a-b300-49de-8994-39e07ce01508\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") "
Sep 16 21:55:26.551052 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.550994    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8fb9ef5a-b300-49de-8994-39e07ce01508" (UID: "8fb9ef5a-b300-49de-8994-39e07ce01508"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:55:26.551369 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.551052    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb9ef5a-b300-49de-8994-39e07ce01508-istiod-ca-cert\") pod \"8fb9ef5a-b300-49de-8994-39e07ce01508\" (UID: \"8fb9ef5a-b300-49de-8994-39e07ce01508\") "
Sep 16 21:55:26.551369 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.551107    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8fb9ef5a-b300-49de-8994-39e07ce01508" (UID: "8fb9ef5a-b300-49de-8994-39e07ce01508"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:55:26.551369 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.551190    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8fb9ef5a-b300-49de-8994-39e07ce01508" (UID: "8fb9ef5a-b300-49de-8994-39e07ce01508"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:55:26.551369 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.551216    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-data" (OuterVolumeSpecName: "istio-data") pod "8fb9ef5a-b300-49de-8994-39e07ce01508" (UID: "8fb9ef5a-b300-49de-8994-39e07ce01508"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:55:26.551578 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.551436    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:26.551578 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.551457    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:26.551578 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.551461    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb9ef5a-b300-49de-8994-39e07ce01508-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8fb9ef5a-b300-49de-8994-39e07ce01508" (UID: "8fb9ef5a-b300-49de-8994-39e07ce01508"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:55:26.551578 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.551471    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:26.551578 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.551504    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:26.553183 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.553149    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8fb9ef5a-b300-49de-8994-39e07ce01508" (UID: "8fb9ef5a-b300-49de-8994-39e07ce01508"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:55:26.553317 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.553176    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb9ef5a-b300-49de-8994-39e07ce01508-kube-api-access-mftbb" (OuterVolumeSpecName: "kube-api-access-mftbb") pod "8fb9ef5a-b300-49de-8994-39e07ce01508" (UID: "8fb9ef5a-b300-49de-8994-39e07ce01508"). InnerVolumeSpecName "kube-api-access-mftbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:55:26.553317 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.553149    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8fb9ef5a-b300-49de-8994-39e07ce01508" (UID: "8fb9ef5a-b300-49de-8994-39e07ce01508"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:55:26.553497 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.553480    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-token" (OuterVolumeSpecName: "istio-token") pod "8fb9ef5a-b300-49de-8994-39e07ce01508" (UID: "8fb9ef5a-b300-49de-8994-39e07ce01508"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:55:26.652635 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.652601    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:26.652635 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.652628    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb9ef5a-b300-49de-8994-39e07ce01508-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:26.652635 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.652636    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:26.652885 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.652649    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mftbb\" (UniqueName: \"kubernetes.io/projected/8fb9ef5a-b300-49de-8994-39e07ce01508-kube-api-access-mftbb\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:26.652885 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:26.652675    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb9ef5a-b300-49de-8994-39e07ce01508-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:27.176689 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:27.176634    2563 generic.go:356] "Generic (PLEG): container finished" podID="8fb9ef5a-b300-49de-8994-39e07ce01508" containerID="b7f3df3b4b2a19b4974a9536d4042196420017f1a51ac010902ed43ca8c7b23c" exitCode=0
Sep 16 21:55:27.176872 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:27.176694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t" event={"ID":"8fb9ef5a-b300-49de-8994-39e07ce01508","Type":"ContainerDied","Data":"b7f3df3b4b2a19b4974a9536d4042196420017f1a51ac010902ed43ca8c7b23c"}
Sep 16 21:55:27.176872 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:27.176751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t" event={"ID":"8fb9ef5a-b300-49de-8994-39e07ce01508","Type":"ContainerDied","Data":"1210f57a5e90f017c10d849a814bc7d568770987f4189047da4ba4ec332495bd"}
Sep 16 21:55:27.176872 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:27.176774    2563 scope.go:117] "RemoveContainer" containerID="b7f3df3b4b2a19b4974a9536d4042196420017f1a51ac010902ed43ca8c7b23c"
Sep 16 21:55:27.176872 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:27.176704    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"
Sep 16 21:55:27.185800 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:27.185780    2563 scope.go:117] "RemoveContainer" containerID="5f5d3140496b8c8119b4316029763adc3e2343b391104dac3f83b68fd73167c1"
Sep 16 21:55:27.192819 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:27.192787    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"]
Sep 16 21:55:27.194671 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:27.194639    2563 scope.go:117] "RemoveContainer" containerID="b7f3df3b4b2a19b4974a9536d4042196420017f1a51ac010902ed43ca8c7b23c"
Sep 16 21:55:27.195037 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:55:27.195014    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f3df3b4b2a19b4974a9536d4042196420017f1a51ac010902ed43ca8c7b23c\": container with ID starting with b7f3df3b4b2a19b4974a9536d4042196420017f1a51ac010902ed43ca8c7b23c not found: ID does not exist" containerID="b7f3df3b4b2a19b4974a9536d4042196420017f1a51ac010902ed43ca8c7b23c"
Sep 16 21:55:27.195141 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:27.195048    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f3df3b4b2a19b4974a9536d4042196420017f1a51ac010902ed43ca8c7b23c"} err="failed to get container status \"b7f3df3b4b2a19b4974a9536d4042196420017f1a51ac010902ed43ca8c7b23c\": rpc error: code = NotFound desc = could not find container \"b7f3df3b4b2a19b4974a9536d4042196420017f1a51ac010902ed43ca8c7b23c\": container with ID starting with b7f3df3b4b2a19b4974a9536d4042196420017f1a51ac010902ed43ca8c7b23c not found: ID does not exist"
Sep 16 21:55:27.195141 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:27.195073    2563 scope.go:117] "RemoveContainer" containerID="5f5d3140496b8c8119b4316029763adc3e2343b391104dac3f83b68fd73167c1"
Sep 16 21:55:27.195414 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:55:27.195395    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f5d3140496b8c8119b4316029763adc3e2343b391104dac3f83b68fd73167c1\": container with ID starting with 5f5d3140496b8c8119b4316029763adc3e2343b391104dac3f83b68fd73167c1 not found: ID does not exist" containerID="5f5d3140496b8c8119b4316029763adc3e2343b391104dac3f83b68fd73167c1"
Sep 16 21:55:27.195455 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:27.195426    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f5d3140496b8c8119b4316029763adc3e2343b391104dac3f83b68fd73167c1"} err="failed to get container status \"5f5d3140496b8c8119b4316029763adc3e2343b391104dac3f83b68fd73167c1\": rpc error: code = NotFound desc = could not find container \"5f5d3140496b8c8119b4316029763adc3e2343b391104dac3f83b68fd73167c1\": container with ID starting with 5f5d3140496b8c8119b4316029763adc3e2343b391104dac3f83b68fd73167c1 not found: ID does not exist"
Sep 16 21:55:27.196220 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:27.196201    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a7df81-openshift-default-7b5ccb5d74-mqs2t"]
Sep 16 21:55:28.608744 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:28.608709    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fb9ef5a-b300-49de-8994-39e07ce01508" path="/var/lib/kubelet/pods/8fb9ef5a-b300-49de-8994-39e07ce01508/volumes"
Sep 16 21:55:32.231015 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:32.230982    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc"]
Sep 16 21:55:32.233611 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:32.233583    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-1b68db-nwkbc"]
Sep 16 21:55:32.607875 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:32.607791    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83490087-7dfd-4fa8-9781-51ddc978cc20" path="/var/lib/kubelet/pods/83490087-7dfd-4fa8-9781-51ddc978cc20/volumes"
Sep 16 21:55:38.799179 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:38.799140    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"]
Sep 16 21:55:38.799707 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:38.799645    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4" podUID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerName="istio-proxy" containerID="cri-o://385f1e757e30eb02daed326d2bdbc9ba6e4a66e2b9b312555513946876f938f8" gracePeriod=30
Sep 16 21:55:41.994210 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:41.994169    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 21:55:41.994593 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:41.994230    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4" podUID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 21:55:44.052446 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.052418    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:55:44.120626 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.120519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-credential-socket\") pod \"49ea78ff-4414-4ffa-9791-8248d6f43535\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") "
Sep 16 21:55:44.120626 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.120564    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7wq8\" (UniqueName: \"kubernetes.io/projected/49ea78ff-4414-4ffa-9791-8248d6f43535-kube-api-access-p7wq8\") pod \"49ea78ff-4414-4ffa-9791-8248d6f43535\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") "
Sep 16 21:55:44.120626 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.120584    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ea78ff-4414-4ffa-9791-8248d6f43535-istiod-ca-cert\") pod \"49ea78ff-4414-4ffa-9791-8248d6f43535\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") "
Sep 16 21:55:44.120626 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.120618    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-data\") pod \"49ea78ff-4414-4ffa-9791-8248d6f43535\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") "
Sep 16 21:55:44.121026 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.120641    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-token\") pod \"49ea78ff-4414-4ffa-9791-8248d6f43535\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") "
Sep 16 21:55:44.121026 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.120704    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-workload-certs\") pod \"49ea78ff-4414-4ffa-9791-8248d6f43535\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") "
Sep 16 21:55:44.121026 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.120763    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-workload-socket\") pod \"49ea78ff-4414-4ffa-9791-8248d6f43535\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") "
Sep 16 21:55:44.121026 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.120859    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-envoy\") pod \"49ea78ff-4414-4ffa-9791-8248d6f43535\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") "
Sep 16 21:55:44.121026 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.120873    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "49ea78ff-4414-4ffa-9791-8248d6f43535" (UID: "49ea78ff-4414-4ffa-9791-8248d6f43535"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:55:44.121026 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.120912    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-podinfo\") pod \"49ea78ff-4414-4ffa-9791-8248d6f43535\" (UID: \"49ea78ff-4414-4ffa-9791-8248d6f43535\") "
Sep 16 21:55:44.121367 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.121048    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-data" (OuterVolumeSpecName: "istio-data") pod "49ea78ff-4414-4ffa-9791-8248d6f43535" (UID: "49ea78ff-4414-4ffa-9791-8248d6f43535"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:55:44.121367 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.121193    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "49ea78ff-4414-4ffa-9791-8248d6f43535" (UID: "49ea78ff-4414-4ffa-9791-8248d6f43535"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:55:44.121367 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.121253    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-credential-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:44.121367 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.121271    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-data\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:44.121367 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.121286    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-workload-socket\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:44.121367 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.121322    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "49ea78ff-4414-4ffa-9791-8248d6f43535" (UID: "49ea78ff-4414-4ffa-9791-8248d6f43535"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:55:44.121731 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.121401    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ea78ff-4414-4ffa-9791-8248d6f43535-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "49ea78ff-4414-4ffa-9791-8248d6f43535" (UID: "49ea78ff-4414-4ffa-9791-8248d6f43535"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:55:44.123586 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.123554    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "49ea78ff-4414-4ffa-9791-8248d6f43535" (UID: "49ea78ff-4414-4ffa-9791-8248d6f43535"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:55:44.123745 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.123708    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ea78ff-4414-4ffa-9791-8248d6f43535-kube-api-access-p7wq8" (OuterVolumeSpecName: "kube-api-access-p7wq8") pod "49ea78ff-4414-4ffa-9791-8248d6f43535" (UID: "49ea78ff-4414-4ffa-9791-8248d6f43535"). InnerVolumeSpecName "kube-api-access-p7wq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:55:44.123992 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.123956    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "49ea78ff-4414-4ffa-9791-8248d6f43535" (UID: "49ea78ff-4414-4ffa-9791-8248d6f43535"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 21:55:44.124212 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.124190    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-token" (OuterVolumeSpecName: "istio-token") pod "49ea78ff-4414-4ffa-9791-8248d6f43535" (UID: "49ea78ff-4414-4ffa-9791-8248d6f43535"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:55:44.222005 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.221950    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-envoy\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:44.222005 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.221997    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-podinfo\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:44.222005 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.222013    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7wq8\" (UniqueName: \"kubernetes.io/projected/49ea78ff-4414-4ffa-9791-8248d6f43535-kube-api-access-p7wq8\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:44.222350 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.222028    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49ea78ff-4414-4ffa-9791-8248d6f43535-istiod-ca-cert\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:44.222350 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.222043    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49ea78ff-4414-4ffa-9791-8248d6f43535-istio-token\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:44.222350 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.222056    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49ea78ff-4414-4ffa-9791-8248d6f43535-workload-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 21:55:44.258489 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.258456    2563 generic.go:356] "Generic (PLEG): container finished" podID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerID="385f1e757e30eb02daed326d2bdbc9ba6e4a66e2b9b312555513946876f938f8" exitCode=0
Sep 16 21:55:44.258689 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.258505    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"
Sep 16 21:55:44.258689 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.258521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4" event={"ID":"49ea78ff-4414-4ffa-9791-8248d6f43535","Type":"ContainerDied","Data":"385f1e757e30eb02daed326d2bdbc9ba6e4a66e2b9b312555513946876f938f8"}
Sep 16 21:55:44.258689 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.258591    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4" event={"ID":"49ea78ff-4414-4ffa-9791-8248d6f43535","Type":"ContainerDied","Data":"663650b6b2ece1d89728aa08be61194297c74f32bd3d2c4ce9012e3ab7ca01cd"}
Sep 16 21:55:44.258689 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.258613    2563 scope.go:117] "RemoveContainer" containerID="385f1e757e30eb02daed326d2bdbc9ba6e4a66e2b9b312555513946876f938f8"
Sep 16 21:55:44.267984 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.267961    2563 scope.go:117] "RemoveContainer" containerID="5ea55bab54e14c067af4c5d52963ca415eba38d93745ecc6e923f81082de5650"
Sep 16 21:55:44.276592 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.276560    2563 scope.go:117] "RemoveContainer" containerID="385f1e757e30eb02daed326d2bdbc9ba6e4a66e2b9b312555513946876f938f8"
Sep 16 21:55:44.276983 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:55:44.276960    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385f1e757e30eb02daed326d2bdbc9ba6e4a66e2b9b312555513946876f938f8\": container with ID starting with 385f1e757e30eb02daed326d2bdbc9ba6e4a66e2b9b312555513946876f938f8 not found: ID does not exist" containerID="385f1e757e30eb02daed326d2bdbc9ba6e4a66e2b9b312555513946876f938f8"
Sep 16 21:55:44.277070 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.276990    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385f1e757e30eb02daed326d2bdbc9ba6e4a66e2b9b312555513946876f938f8"} err="failed to get container status \"385f1e757e30eb02daed326d2bdbc9ba6e4a66e2b9b312555513946876f938f8\": rpc error: code = NotFound desc = could not find container \"385f1e757e30eb02daed326d2bdbc9ba6e4a66e2b9b312555513946876f938f8\": container with ID starting with 385f1e757e30eb02daed326d2bdbc9ba6e4a66e2b9b312555513946876f938f8 not found: ID does not exist"
Sep 16 21:55:44.277070 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.277009    2563 scope.go:117] "RemoveContainer" containerID="5ea55bab54e14c067af4c5d52963ca415eba38d93745ecc6e923f81082de5650"
Sep 16 21:55:44.277332 ip-10-0-136-118 kubenswrapper[2563]: E0916 21:55:44.277306    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea55bab54e14c067af4c5d52963ca415eba38d93745ecc6e923f81082de5650\": container with ID starting with 5ea55bab54e14c067af4c5d52963ca415eba38d93745ecc6e923f81082de5650 not found: ID does not exist" containerID="5ea55bab54e14c067af4c5d52963ca415eba38d93745ecc6e923f81082de5650"
Sep 16 21:55:44.277426 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.277341    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea55bab54e14c067af4c5d52963ca415eba38d93745ecc6e923f81082de5650"} err="failed to get container status \"5ea55bab54e14c067af4c5d52963ca415eba38d93745ecc6e923f81082de5650\": rpc error: code = NotFound desc = could not find container \"5ea55bab54e14c067af4c5d52963ca415eba38d93745ecc6e923f81082de5650\": container with ID starting with 5ea55bab54e14c067af4c5d52963ca415eba38d93745ecc6e923f81082de5650 not found: ID does not exist"
Sep 16 21:55:44.281268 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.281188    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"]
Sep 16 21:55:44.284925 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.284904    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1b68db-openshift-default-7496b948dd-5j9m4"]
Sep 16 21:55:44.608085 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:55:44.608054    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49ea78ff-4414-4ffa-9791-8248d6f43535" path="/var/lib/kubelet/pods/49ea78ff-4414-4ffa-9791-8248d6f43535/volumes"
Sep 16 21:59:21.192160 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:21.192122    2563 scope.go:117] "RemoveContainer" containerID="2ffc835c81f34a0a4e347fc8dd5c0f0c748f347ee7896617c222fdd47da261be"
Sep 16 21:59:59.578378 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578300    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"]
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578634    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fb9ef5a-b300-49de-8994-39e07ce01508" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578645    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fb9ef5a-b300-49de-8994-39e07ce01508" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578672    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57903415-58fe-452e-ae2a-eb30704bcff0" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578680    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="57903415-58fe-452e-ae2a-eb30704bcff0" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578699    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c656493-fbbe-4f82-82c2-c56a0319e728" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578705    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c656493-fbbe-4f82-82c2-c56a0319e728" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578713    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fb9ef5a-b300-49de-8994-39e07ce01508" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578718    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fb9ef5a-b300-49de-8994-39e07ce01508" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578724    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578729    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578740    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83490087-7dfd-4fa8-9781-51ddc978cc20" containerName="revoke-keys"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578746    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="83490087-7dfd-4fa8-9781-51ddc978cc20" containerName="revoke-keys"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578752    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578757    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6978ea7c-6aae-4c24-adf9-6a1498e9aabc" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578815    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fb9ef5a-b300-49de-8994-39e07ce01508" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578823    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578830    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578837    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c656493-fbbe-4f82-82c2-c56a0319e728" containerName="istio-proxy"
Sep 16 21:59:59.578840 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.578843    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="83490087-7dfd-4fa8-9781-51ddc978cc20" containerName="revoke-keys"
Sep 16 21:59:59.581821 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.581805    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.584026 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.584007    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-44cf9098-kserve-self-signed-certs\""
Sep 16 21:59:59.590593 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.590567    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"]
Sep 16 21:59:59.592814 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.592788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-model-cache\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.592924 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.592819    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-home\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.592924 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.592848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-tls-certs\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.593002 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.592919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-dshm\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.593002 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.592994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxhdv\" (UniqueName: \"kubernetes.io/projected/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-kube-api-access-qxhdv\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.593068 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.593025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-tmp-dir\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.694376 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.694346    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxhdv\" (UniqueName: \"kubernetes.io/projected/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-kube-api-access-qxhdv\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.694555 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.694387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-tmp-dir\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.694555 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.694415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-model-cache\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.694555 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.694525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-home\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.694770 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.694591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-tls-certs\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.694770 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.694630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-dshm\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.694885 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.694778    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-model-cache\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.694885 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.694807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-tmp-dir\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.694983 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.694920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-home\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.696933 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.696907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-dshm\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.697072 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.697053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-tls-certs\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.702782 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.702766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxhdv\" (UniqueName: \"kubernetes.io/projected/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-kube-api-access-qxhdv\") pod \"e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") " pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 21:59:59.891873 ip-10-0-136-118 kubenswrapper[2563]: I0916 21:59:59.891783    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 22:00:00.129275 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:00.129239    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826600-7d5jr"]
Sep 16 22:00:00.129622 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:00.129609    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerName="istio-proxy"
Sep 16 22:00:00.129704 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:00.129624    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ea78ff-4414-4ffa-9791-8248d6f43535" containerName="istio-proxy"
Sep 16 22:00:00.129704 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:00.129700    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fb9ef5a-b300-49de-8994-39e07ce01508" containerName="istio-proxy"
Sep 16 22:00:00.132792 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:00.132774    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826600-7d5jr"
Sep 16 22:00:00.134815 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:00.134796    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jqx5l\""
Sep 16 22:00:00.140738 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:00.140715    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826600-7d5jr"]
Sep 16 22:00:00.143440 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:00.143396    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826600-7d5jr"
Sep 16 22:00:00.184459 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:00.184117    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"]
Sep 16 22:00:00.186050 ip-10-0-136-118 kubenswrapper[2563]: W0916 22:00:00.186023    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe6d412_58c9_4afd_aa51_46dfc3fbfde5.slice/crio-3db3de80cb14609408ffca6bba1f7deb884be0792102e3ba6ff8b2db0eb2898a WatchSource:0}: Error finding container 3db3de80cb14609408ffca6bba1f7deb884be0792102e3ba6ff8b2db0eb2898a: Status 404 returned error can't find the container with id 3db3de80cb14609408ffca6bba1f7deb884be0792102e3ba6ff8b2db0eb2898a
Sep 16 22:00:00.409279 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:00.409190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt" event={"ID":"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5","Type":"ContainerStarted","Data":"2523af5f9d0737b4624bc4083ba1c15fc9d79d0915af336c8a3a3fe0b227f868"}
Sep 16 22:00:00.409279 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:00.409243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt" event={"ID":"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5","Type":"ContainerStarted","Data":"3db3de80cb14609408ffca6bba1f7deb884be0792102e3ba6ff8b2db0eb2898a"}
Sep 16 22:00:00.416239 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:00.416216    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826600-7d5jr"]
Sep 16 22:00:00.418498 ip-10-0-136-118 kubenswrapper[2563]: W0916 22:00:00.418477    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d4bef0_6886_448e_85da_c17cf4ae85d9.slice/crio-6e9c2e8edf7f4197dfd4a468365ed087097bab6fd7bcd095780c605ee530f3cc WatchSource:0}: Error finding container 6e9c2e8edf7f4197dfd4a468365ed087097bab6fd7bcd095780c605ee530f3cc: Status 404 returned error can't find the container with id 6e9c2e8edf7f4197dfd4a468365ed087097bab6fd7bcd095780c605ee530f3cc
Sep 16 22:00:00.426604 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:00.426561    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt" podStartSLOduration=1.426542587 podStartE2EDuration="1.426542587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 22:00:00.425832587 +0000 UTC m=+2260.326042582" watchObservedRunningTime="2026-09-16 22:00:00.426542587 +0000 UTC m=+2260.326752592"
Sep 16 22:00:01.415222 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:01.415188    2563 generic.go:356] "Generic (PLEG): container finished" podID="63d4bef0-6886-448e-85da-c17cf4ae85d9" containerID="78e8cb87bf0fef8233c81a8a36923706675f93326b00c1e2f5b1817df5f32ade" exitCode=0
Sep 16 22:00:01.415592 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:01.415273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826600-7d5jr" event={"ID":"63d4bef0-6886-448e-85da-c17cf4ae85d9","Type":"ContainerDied","Data":"78e8cb87bf0fef8233c81a8a36923706675f93326b00c1e2f5b1817df5f32ade"}
Sep 16 22:00:01.415592 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:01.415327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826600-7d5jr" event={"ID":"63d4bef0-6886-448e-85da-c17cf4ae85d9","Type":"ContainerStarted","Data":"6e9c2e8edf7f4197dfd4a468365ed087097bab6fd7bcd095780c605ee530f3cc"}
Sep 16 22:00:02.550522 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:02.550490    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826600-7d5jr"
Sep 16 22:00:03.427528 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:03.427486    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826600-7d5jr" event={"ID":"63d4bef0-6886-448e-85da-c17cf4ae85d9","Type":"ContainerDied","Data":"6e9c2e8edf7f4197dfd4a468365ed087097bab6fd7bcd095780c605ee530f3cc"}
Sep 16 22:00:03.427528 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:03.427505    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826600-7d5jr"
Sep 16 22:00:03.427528 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:03.427538    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e9c2e8edf7f4197dfd4a468365ed087097bab6fd7bcd095780c605ee530f3cc"
Sep 16 22:00:09.892314 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:09.892277    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 22:00:09.892802 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:09.892351    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 22:00:09.904894 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:09.904867    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 22:00:10.471988 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:10.471955    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 22:00:21.213017 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:21.212973    2563 scope.go:117] "RemoveContainer" containerID="99e16b0ed0e100fd89134d2d493b3aa594c3ae3eccdc5b2104e32ee943d8068e"
Sep 16 22:00:27.991246 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:27.991210    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"]
Sep 16 22:00:43.012617 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.012575    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt" podUID="1fe6d412-58c9-4afd-aa51-46dfc3fbfde5" containerName="main" containerID="cri-o://2523af5f9d0737b4624bc4083ba1c15fc9d79d0915af336c8a3a3fe0b227f868" gracePeriod=45
Sep 16 22:00:43.261974 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.261948    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 22:00:43.305104 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.305021    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"]
Sep 16 22:00:43.305359 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.305347    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe6d412-58c9-4afd-aa51-46dfc3fbfde5" containerName="main"
Sep 16 22:00:43.305407 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.305360    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe6d412-58c9-4afd-aa51-46dfc3fbfde5" containerName="main"
Sep 16 22:00:43.305407 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.305380    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63d4bef0-6886-448e-85da-c17cf4ae85d9" containerName="cleanup"
Sep 16 22:00:43.305407 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.305385    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="63d4bef0-6886-448e-85da-c17cf4ae85d9" containerName="cleanup"
Sep 16 22:00:43.305519 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.305453    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe6d412-58c9-4afd-aa51-46dfc3fbfde5" containerName="main"
Sep 16 22:00:43.305519 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.305465    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="63d4bef0-6886-448e-85da-c17cf4ae85d9" containerName="cleanup"
Sep 16 22:00:43.308404 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.308383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:43.310735 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.310632    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 22:00:43.311108 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.310953    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lnht4\""
Sep 16 22:00:43.311108 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.311029    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 22:00:43.311108 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.311093    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 22:00:43.311308 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.311235    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 22:00:43.320358 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.320206    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"]
Sep 16 22:00:43.320505 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.320405    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-dshm\") pod \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") "
Sep 16 22:00:43.320505 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.320457    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-home\") pod \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") "
Sep 16 22:00:43.320505 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.320490    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-model-cache\") pod \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") "
Sep 16 22:00:43.320706 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.320519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-tmp-dir\") pod \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") "
Sep 16 22:00:43.320706 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.320645    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxhdv\" (UniqueName: \"kubernetes.io/projected/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-kube-api-access-qxhdv\") pod \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") "
Sep 16 22:00:43.320706 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.320700    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-tls-certs\") pod \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\" (UID: \"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5\") "
Sep 16 22:00:43.320924 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.320882    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-model-cache" (OuterVolumeSpecName: "model-cache") pod "1fe6d412-58c9-4afd-aa51-46dfc3fbfde5" (UID: "1fe6d412-58c9-4afd-aa51-46dfc3fbfde5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 22:00:43.320924 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.320892    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1fe6d412-58c9-4afd-aa51-46dfc3fbfde5" (UID: "1fe6d412-58c9-4afd-aa51-46dfc3fbfde5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 22:00:43.320924 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.320906    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-home" (OuterVolumeSpecName: "home") pod "1fe6d412-58c9-4afd-aa51-46dfc3fbfde5" (UID: "1fe6d412-58c9-4afd-aa51-46dfc3fbfde5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 22:00:43.321184 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.321022    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-model-cache\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 22:00:43.321184 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.321042    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-tmp-dir\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 22:00:43.321184 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.321054    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-home\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 22:00:43.323349 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.323326    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-dshm" (OuterVolumeSpecName: "dshm") pod "1fe6d412-58c9-4afd-aa51-46dfc3fbfde5" (UID: "1fe6d412-58c9-4afd-aa51-46dfc3fbfde5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 22:00:43.323509 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.323475    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-kube-api-access-qxhdv" (OuterVolumeSpecName: "kube-api-access-qxhdv") pod "1fe6d412-58c9-4afd-aa51-46dfc3fbfde5" (UID: "1fe6d412-58c9-4afd-aa51-46dfc3fbfde5"). InnerVolumeSpecName "kube-api-access-qxhdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 22:00:43.324033 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.324008    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1fe6d412-58c9-4afd-aa51-46dfc3fbfde5" (UID: "1fe6d412-58c9-4afd-aa51-46dfc3fbfde5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 22:00:43.422074 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.422029    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-openshift-service-ca\") pod \"maas-api-7d747b4f9d-xs75x\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:43.422266 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.422083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw9pf\" (UniqueName: \"kubernetes.io/projected/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-kube-api-access-bw9pf\") pod \"maas-api-7d747b4f9d-xs75x\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:43.422266 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.422169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-metrics-certs\") pod \"maas-api-7d747b4f9d-xs75x\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:43.422266 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.422215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-maas-api-tls\") pod \"maas-api-7d747b4f9d-xs75x\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:43.422461 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.422290    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxhdv\" (UniqueName: \"kubernetes.io/projected/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-kube-api-access-qxhdv\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 22:00:43.422461 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.422312    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-tls-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 22:00:43.422461 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.422322    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5-dshm\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 22:00:43.523396 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.523353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw9pf\" (UniqueName: \"kubernetes.io/projected/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-kube-api-access-bw9pf\") pod \"maas-api-7d747b4f9d-xs75x\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:43.523616 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.523402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-metrics-certs\") pod \"maas-api-7d747b4f9d-xs75x\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:43.523616 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.523437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-maas-api-tls\") pod \"maas-api-7d747b4f9d-xs75x\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:43.523616 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.523508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-openshift-service-ca\") pod \"maas-api-7d747b4f9d-xs75x\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:43.524222 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.524196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-openshift-service-ca\") pod \"maas-api-7d747b4f9d-xs75x\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:43.525991 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.525967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-maas-api-tls\") pod \"maas-api-7d747b4f9d-xs75x\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:43.526078 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.526007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-metrics-certs\") pod \"maas-api-7d747b4f9d-xs75x\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:43.530929 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.530911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw9pf\" (UniqueName: \"kubernetes.io/projected/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-kube-api-access-bw9pf\") pod \"maas-api-7d747b4f9d-xs75x\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:43.615961 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.615866    2563 generic.go:356] "Generic (PLEG): container finished" podID="1fe6d412-58c9-4afd-aa51-46dfc3fbfde5" containerID="2523af5f9d0737b4624bc4083ba1c15fc9d79d0915af336c8a3a3fe0b227f868" exitCode=0
Sep 16 22:00:43.615961 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.615913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt" event={"ID":"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5","Type":"ContainerDied","Data":"2523af5f9d0737b4624bc4083ba1c15fc9d79d0915af336c8a3a3fe0b227f868"}
Sep 16 22:00:43.615961 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.615916    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"
Sep 16 22:00:43.615961 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.615944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt" event={"ID":"1fe6d412-58c9-4afd-aa51-46dfc3fbfde5","Type":"ContainerDied","Data":"3db3de80cb14609408ffca6bba1f7deb884be0792102e3ba6ff8b2db0eb2898a"}
Sep 16 22:00:43.615961 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.615960    2563 scope.go:117] "RemoveContainer" containerID="2523af5f9d0737b4624bc4083ba1c15fc9d79d0915af336c8a3a3fe0b227f868"
Sep 16 22:00:43.619335 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.619317    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:43.625101 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.625077    2563 scope.go:117] "RemoveContainer" containerID="2523af5f9d0737b4624bc4083ba1c15fc9d79d0915af336c8a3a3fe0b227f868"
Sep 16 22:00:43.625406 ip-10-0-136-118 kubenswrapper[2563]: E0916 22:00:43.625385    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2523af5f9d0737b4624bc4083ba1c15fc9d79d0915af336c8a3a3fe0b227f868\": container with ID starting with 2523af5f9d0737b4624bc4083ba1c15fc9d79d0915af336c8a3a3fe0b227f868 not found: ID does not exist" containerID="2523af5f9d0737b4624bc4083ba1c15fc9d79d0915af336c8a3a3fe0b227f868"
Sep 16 22:00:43.625494 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.625412    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2523af5f9d0737b4624bc4083ba1c15fc9d79d0915af336c8a3a3fe0b227f868"} err="failed to get container status \"2523af5f9d0737b4624bc4083ba1c15fc9d79d0915af336c8a3a3fe0b227f868\": rpc error: code = NotFound desc = could not find container \"2523af5f9d0737b4624bc4083ba1c15fc9d79d0915af336c8a3a3fe0b227f868\": container with ID starting with 2523af5f9d0737b4624bc4083ba1c15fc9d79d0915af336c8a3a3fe0b227f868 not found: ID does not exist"
Sep 16 22:00:43.637067 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.637043    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"]
Sep 16 22:00:43.640861 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.640836    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-44cf9098-kserve-79b5766b7-555tt"]
Sep 16 22:00:43.893761 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.893705    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"]
Sep 16 22:00:43.895976 ip-10-0-136-118 kubenswrapper[2563]: W0916 22:00:43.895951    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ef0542_32c7_4fb4_8cef_bc9dfa2ab945.slice/crio-f99a7d578bdfa959b69cf63d95ae8d4d07de50120dbdff4eb711af3fbd526114 WatchSource:0}: Error finding container f99a7d578bdfa959b69cf63d95ae8d4d07de50120dbdff4eb711af3fbd526114: Status 404 returned error can't find the container with id f99a7d578bdfa959b69cf63d95ae8d4d07de50120dbdff4eb711af3fbd526114
Sep 16 22:00:43.897288 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:43.897271    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 22:00:44.608035 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:44.607995    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fe6d412-58c9-4afd-aa51-46dfc3fbfde5" path="/var/lib/kubelet/pods/1fe6d412-58c9-4afd-aa51-46dfc3fbfde5/volumes"
Sep 16 22:00:44.621386 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:44.621351    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x" event={"ID":"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945","Type":"ContainerStarted","Data":"f99a7d578bdfa959b69cf63d95ae8d4d07de50120dbdff4eb711af3fbd526114"}
Sep 16 22:00:48.644557 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:48.644518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x" event={"ID":"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945","Type":"ContainerStarted","Data":"8d774e95c9d6a5d246eb7aad16dcc1426427e41ef63049f1ca6fbb8ff2057880"}
Sep 16 22:00:48.645068 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:48.644587    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:48.661830 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:48.661770    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x" podStartSLOduration=1.936298427 podStartE2EDuration="5.66175385s" totalImagesPullingTime="3.725455423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 22:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 22:00:48.659868455 +0000 UTC m=+2308.560078458" watchObservedRunningTime="2026-09-16 22:00:48.66175385 +0000 UTC m=+2308.561963854"
Sep 16 22:00:50.866198 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:50.866159    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"]
Sep 16 22:00:50.866675 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:50.866363    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x" podUID="c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945" containerName="maas-api" containerID="cri-o://8d774e95c9d6a5d246eb7aad16dcc1426427e41ef63049f1ca6fbb8ff2057880" gracePeriod=30
Sep 16 22:00:51.113940 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.113908    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:51.200847 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.200807    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-openshift-service-ca\") pod \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") "
Sep 16 22:00:51.201034 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.200876    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw9pf\" (UniqueName: \"kubernetes.io/projected/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-kube-api-access-bw9pf\") pod \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") "
Sep 16 22:00:51.201034 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.200948    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-metrics-certs\") pod \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") "
Sep 16 22:00:51.201131 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.201038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-maas-api-tls\") pod \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\" (UID: \"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945\") "
Sep 16 22:00:51.201287 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.201261    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945" (UID: "c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 22:00:51.201378 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.201327    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-openshift-service-ca\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 22:00:51.203337 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.203305    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945" (UID: "c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 22:00:51.203337 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.203321    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945" (UID: "c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 22:00:51.203480 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.203320    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-kube-api-access-bw9pf" (OuterVolumeSpecName: "kube-api-access-bw9pf") pod "c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945" (UID: "c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945"). InnerVolumeSpecName "kube-api-access-bw9pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 22:00:51.302723 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.302644    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-maas-api-tls\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 22:00:51.302723 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.302715    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw9pf\" (UniqueName: \"kubernetes.io/projected/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-kube-api-access-bw9pf\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 22:00:51.302723 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.302728    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945-metrics-certs\") on node \"ip-10-0-136-118.ec2.internal\" DevicePath \"\""
Sep 16 22:00:51.659823 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.659721    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945" containerID="8d774e95c9d6a5d246eb7aad16dcc1426427e41ef63049f1ca6fbb8ff2057880" exitCode=0
Sep 16 22:00:51.659823 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.659775    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"
Sep 16 22:00:51.660075 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.659777    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x" event={"ID":"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945","Type":"ContainerDied","Data":"8d774e95c9d6a5d246eb7aad16dcc1426427e41ef63049f1ca6fbb8ff2057880"}
Sep 16 22:00:51.660075 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.659921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x" event={"ID":"c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945","Type":"ContainerDied","Data":"f99a7d578bdfa959b69cf63d95ae8d4d07de50120dbdff4eb711af3fbd526114"}
Sep 16 22:00:51.660075 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.659960    2563 scope.go:117] "RemoveContainer" containerID="8d774e95c9d6a5d246eb7aad16dcc1426427e41ef63049f1ca6fbb8ff2057880"
Sep 16 22:00:51.674783 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.674750    2563 scope.go:117] "RemoveContainer" containerID="8d774e95c9d6a5d246eb7aad16dcc1426427e41ef63049f1ca6fbb8ff2057880"
Sep 16 22:00:51.675198 ip-10-0-136-118 kubenswrapper[2563]: E0916 22:00:51.675171    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d774e95c9d6a5d246eb7aad16dcc1426427e41ef63049f1ca6fbb8ff2057880\": container with ID starting with 8d774e95c9d6a5d246eb7aad16dcc1426427e41ef63049f1ca6fbb8ff2057880 not found: ID does not exist" containerID="8d774e95c9d6a5d246eb7aad16dcc1426427e41ef63049f1ca6fbb8ff2057880"
Sep 16 22:00:51.675287 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.675206    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d774e95c9d6a5d246eb7aad16dcc1426427e41ef63049f1ca6fbb8ff2057880"} err="failed to get container status \"8d774e95c9d6a5d246eb7aad16dcc1426427e41ef63049f1ca6fbb8ff2057880\": rpc error: code = NotFound desc = could not find container \"8d774e95c9d6a5d246eb7aad16dcc1426427e41ef63049f1ca6fbb8ff2057880\": container with ID starting with 8d774e95c9d6a5d246eb7aad16dcc1426427e41ef63049f1ca6fbb8ff2057880 not found: ID does not exist"
Sep 16 22:00:51.683179 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.683146    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"]
Sep 16 22:00:51.686875 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:51.686846    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-xs75x"]
Sep 16 22:00:52.607929 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:00:52.607888    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945" path="/var/lib/kubelet/pods/c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945/volumes"
Sep 16 22:01:12.912457 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:12.912422    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b78788b8b-gp25s_d4bfb8c5-4432-44ec-a798-5abf9401793b/authorino/0.log"
Sep 16 22:01:18.475250 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:18.475218    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bvhxd_7b8b352f-26b5-4ec0-a008-c1519d1ab4c1/server/0.log"
Sep 16 22:01:18.588013 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:18.587983    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-bjt9h_f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b/manager/0.log"
Sep 16 22:01:18.930094 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:18.930061    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df9d657c6-tvj4m_0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0/copy-manifests/0.log"
Sep 16 22:01:18.941898 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:18.941853    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df9d657c6-tvj4m_0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0/manager/0.log"
Sep 16 22:01:19.484610 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:19.484582    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826585-njznj_54596020-9528-40ab-8b32-bb8c9c5b7edb/cleanup/0.log"
Sep 16 22:01:19.591236 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:19.591205    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826600-7d5jr_63d4bef0-6886-448e-85da-c17cf4ae85d9/cleanup/0.log"
Sep 16 22:01:20.447875 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:20.447824    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b78788b8b-gp25s_d4bfb8c5-4432-44ec-a798-5abf9401793b/authorino/0.log"
Sep 16 22:01:20.775442 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:20.775369    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-bbjf4_ed62904c-1291-4169-a246-3e1afc960125/kuadrant-console-plugin/0.log"
Sep 16 22:01:21.455634 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:21.455601    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd_f26de3a9-5d79-44e6-9fae-ebb703fb3020/istio-proxy/0.log"
Sep 16 22:01:21.786789 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:21.786739    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f65bc968c-wszkq_d679b692-a50b-4df9-9f32-42c34a9b58d5/kube-auth-proxy/0.log"
Sep 16 22:01:21.898616 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:21.898585    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bplw6_8ac674de-6254-4602-8a24-e03a8a075fa9/istio-proxy/1.log"
Sep 16 22:01:22.686328 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:22.686300    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-gpg5n_d882622e-983f-4139-9090-ad5df1fd15a0/main/0.log"
Sep 16 22:01:23.222638 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:23.222610    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-t99c6_5ca26678-e9fb-4fd7-a21e-7e49f0a3491f/main/0.log"
Sep 16 22:01:28.321498 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.321455    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pfn8w/must-gather-cdhxr"]
Sep 16 22:01:28.321999 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.321819    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945" containerName="maas-api"
Sep 16 22:01:28.321999 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.321831    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945" containerName="maas-api"
Sep 16 22:01:28.321999 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.321887    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9ef0542-32c7-4fb4-8cef-bc9dfa2ab945" containerName="maas-api"
Sep 16 22:01:28.324963 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.324944    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfn8w/must-gather-cdhxr"
Sep 16 22:01:28.327903 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.327879    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pfn8w\"/\"default-dockercfg-zfj9l\""
Sep 16 22:01:28.328436 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.328414    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfn8w\"/\"kube-root-ca.crt\""
Sep 16 22:01:28.328583 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.328474    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfn8w\"/\"openshift-service-ca.crt\""
Sep 16 22:01:28.342819 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.342785    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfn8w/must-gather-cdhxr"]
Sep 16 22:01:28.440748 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.440706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d51736cf-6684-4bdb-981c-f0a0f62255e6-must-gather-output\") pod \"must-gather-cdhxr\" (UID: \"d51736cf-6684-4bdb-981c-f0a0f62255e6\") " pod="openshift-must-gather-pfn8w/must-gather-cdhxr"
Sep 16 22:01:28.440968 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.440834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcwn7\" (UniqueName: \"kubernetes.io/projected/d51736cf-6684-4bdb-981c-f0a0f62255e6-kube-api-access-hcwn7\") pod \"must-gather-cdhxr\" (UID: \"d51736cf-6684-4bdb-981c-f0a0f62255e6\") " pod="openshift-must-gather-pfn8w/must-gather-cdhxr"
Sep 16 22:01:28.542323 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.542281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcwn7\" (UniqueName: \"kubernetes.io/projected/d51736cf-6684-4bdb-981c-f0a0f62255e6-kube-api-access-hcwn7\") pod \"must-gather-cdhxr\" (UID: \"d51736cf-6684-4bdb-981c-f0a0f62255e6\") " pod="openshift-must-gather-pfn8w/must-gather-cdhxr"
Sep 16 22:01:28.542532 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.542353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d51736cf-6684-4bdb-981c-f0a0f62255e6-must-gather-output\") pod \"must-gather-cdhxr\" (UID: \"d51736cf-6684-4bdb-981c-f0a0f62255e6\") " pod="openshift-must-gather-pfn8w/must-gather-cdhxr"
Sep 16 22:01:28.542707 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.542645    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d51736cf-6684-4bdb-981c-f0a0f62255e6-must-gather-output\") pod \"must-gather-cdhxr\" (UID: \"d51736cf-6684-4bdb-981c-f0a0f62255e6\") " pod="openshift-must-gather-pfn8w/must-gather-cdhxr"
Sep 16 22:01:28.551367 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.551344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcwn7\" (UniqueName: \"kubernetes.io/projected/d51736cf-6684-4bdb-981c-f0a0f62255e6-kube-api-access-hcwn7\") pod \"must-gather-cdhxr\" (UID: \"d51736cf-6684-4bdb-981c-f0a0f62255e6\") " pod="openshift-must-gather-pfn8w/must-gather-cdhxr"
Sep 16 22:01:28.634477 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.634373    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfn8w/must-gather-cdhxr"
Sep 16 22:01:28.907769 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:28.907745    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfn8w/must-gather-cdhxr"]
Sep 16 22:01:28.909999 ip-10-0-136-118 kubenswrapper[2563]: W0916 22:01:28.909971    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd51736cf_6684_4bdb_981c_f0a0f62255e6.slice/crio-c8918241ef23c03c613d0b6a4c4b061d320d1cb4fbbf9416eb46c86a6daa05a6 WatchSource:0}: Error finding container c8918241ef23c03c613d0b6a4c4b061d320d1cb4fbbf9416eb46c86a6daa05a6: Status 404 returned error can't find the container with id c8918241ef23c03c613d0b6a4c4b061d320d1cb4fbbf9416eb46c86a6daa05a6
Sep 16 22:01:29.841361 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:29.841325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfn8w/must-gather-cdhxr" event={"ID":"d51736cf-6684-4bdb-981c-f0a0f62255e6","Type":"ContainerStarted","Data":"c8918241ef23c03c613d0b6a4c4b061d320d1cb4fbbf9416eb46c86a6daa05a6"}
Sep 16 22:01:31.856023 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:31.855977    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfn8w/must-gather-cdhxr" event={"ID":"d51736cf-6684-4bdb-981c-f0a0f62255e6","Type":"ContainerStarted","Data":"0bcd0cdf5a5cb499ae5f43884ab2c631c24d6d9dae32c8b317b648d98d59a2cb"}
Sep 16 22:01:31.856435 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:31.856040    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfn8w/must-gather-cdhxr" event={"ID":"d51736cf-6684-4bdb-981c-f0a0f62255e6","Type":"ContainerStarted","Data":"ee2834a474e73599033b1a6fc74094c7cdb150ae9fd428ab2975ce98a013aa81"}
Sep 16 22:01:31.872153 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:31.872085    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pfn8w/must-gather-cdhxr" podStartSLOduration=1.829346867 podStartE2EDuration="3.872064618s" totalImagesPullingTime="2.042717751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 22:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 22:01:31.86986241 +0000 UTC m=+2351.770072415" watchObservedRunningTime="2026-09-16 22:01:31.872064618 +0000 UTC m=+2351.772274622"
Sep 16 22:01:32.580557 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:32.580520    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zxtz8_8800530f-c408-44d0-a430-bd6d8cd8d95c/global-pull-secret-syncer/0.log"
Sep 16 22:01:32.705893 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:32.705864    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s89hp_371857b4-159c-47eb-897c-e71bb61469c7/konnectivity-agent/0.log"
Sep 16 22:01:32.752701 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:32.752671    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-118.ec2.internal_126583346e4bbf13f4d978c46ad24fd3/haproxy/0.log"
Sep 16 22:01:37.893253 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:37.892922    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b78788b8b-gp25s_d4bfb8c5-4432-44ec-a798-5abf9401793b/authorino/0.log"
Sep 16 22:01:37.983443 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:37.983411    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-bbjf4_ed62904c-1291-4169-a246-3e1afc960125/kuadrant-console-plugin/0.log"
Sep 16 22:01:40.240641 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.240606    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqsbg_59972e1c-28dd-4a1e-8163-66defade025a/node-exporter/0.log"
Sep 16 22:01:40.260085 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.260052    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqsbg_59972e1c-28dd-4a1e-8163-66defade025a/kube-rbac-proxy/0.log"
Sep 16 22:01:40.288542 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.288514    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqsbg_59972e1c-28dd-4a1e-8163-66defade025a/init-textfile/0.log"
Sep 16 22:01:40.322736 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.322639    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-dn5cq_654340c3-d71e-4069-9fff-78a36f7cc28a/kube-rbac-proxy-main/0.log"
Sep 16 22:01:40.344899 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.344871    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-dn5cq_654340c3-d71e-4069-9fff-78a36f7cc28a/kube-rbac-proxy-self/0.log"
Sep 16 22:01:40.366771 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.366733    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-dn5cq_654340c3-d71e-4069-9fff-78a36f7cc28a/openshift-state-metrics/0.log"
Sep 16 22:01:40.409323 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.409234    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b1beb123-dc5e-435f-a028-81b28c797d82/prometheus/0.log"
Sep 16 22:01:40.431451 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.431419    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b1beb123-dc5e-435f-a028-81b28c797d82/config-reloader/0.log"
Sep 16 22:01:40.454130 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.454096    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b1beb123-dc5e-435f-a028-81b28c797d82/thanos-sidecar/0.log"
Sep 16 22:01:40.481613 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.481577    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b1beb123-dc5e-435f-a028-81b28c797d82/kube-rbac-proxy-web/0.log"
Sep 16 22:01:40.502218 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.502181    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b1beb123-dc5e-435f-a028-81b28c797d82/kube-rbac-proxy/0.log"
Sep 16 22:01:40.529916 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.529885    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b1beb123-dc5e-435f-a028-81b28c797d82/kube-rbac-proxy-thanos/0.log"
Sep 16 22:01:40.552904 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.552865    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b1beb123-dc5e-435f-a028-81b28c797d82/init-config-reloader/0.log"
Sep 16 22:01:40.681352 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.681102    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-db94579bd-tldhc_a1793a6c-4e04-461f-b8dd-d557ef233005/telemeter-client/0.log"
Sep 16 22:01:40.705522 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.705493    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-db94579bd-tldhc_a1793a6c-4e04-461f-b8dd-d557ef233005/reload/0.log"
Sep 16 22:01:40.726313 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:40.726250    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-db94579bd-tldhc_a1793a6c-4e04-461f-b8dd-d557ef233005/kube-rbac-proxy/0.log"
Sep 16 22:01:42.986726 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:42.986685    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"]
Sep 16 22:01:42.992547 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:42.992513    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.000275 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.000242    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"]
Sep 16 22:01:43.106832 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.106788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ece8d30-b5c4-4cfa-81b0-49e687d53039-sys\") pod \"perf-node-gather-daemonset-7fv25\" (UID: \"4ece8d30-b5c4-4cfa-81b0-49e687d53039\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.107029 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.106841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4ece8d30-b5c4-4cfa-81b0-49e687d53039-proc\") pod \"perf-node-gather-daemonset-7fv25\" (UID: \"4ece8d30-b5c4-4cfa-81b0-49e687d53039\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.107029 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.106877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ece8d30-b5c4-4cfa-81b0-49e687d53039-lib-modules\") pod \"perf-node-gather-daemonset-7fv25\" (UID: \"4ece8d30-b5c4-4cfa-81b0-49e687d53039\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.107029 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.106972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4ece8d30-b5c4-4cfa-81b0-49e687d53039-podres\") pod \"perf-node-gather-daemonset-7fv25\" (UID: \"4ece8d30-b5c4-4cfa-81b0-49e687d53039\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.107029 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.107006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h25l\" (UniqueName: \"kubernetes.io/projected/4ece8d30-b5c4-4cfa-81b0-49e687d53039-kube-api-access-9h25l\") pod \"perf-node-gather-daemonset-7fv25\" (UID: \"4ece8d30-b5c4-4cfa-81b0-49e687d53039\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.208199 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.208157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ece8d30-b5c4-4cfa-81b0-49e687d53039-sys\") pod \"perf-node-gather-daemonset-7fv25\" (UID: \"4ece8d30-b5c4-4cfa-81b0-49e687d53039\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.208475 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.208452    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4ece8d30-b5c4-4cfa-81b0-49e687d53039-proc\") pod \"perf-node-gather-daemonset-7fv25\" (UID: \"4ece8d30-b5c4-4cfa-81b0-49e687d53039\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.208616 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.208601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ece8d30-b5c4-4cfa-81b0-49e687d53039-lib-modules\") pod \"perf-node-gather-daemonset-7fv25\" (UID: \"4ece8d30-b5c4-4cfa-81b0-49e687d53039\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.208866 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.208831    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ece8d30-b5c4-4cfa-81b0-49e687d53039-sys\") pod \"perf-node-gather-daemonset-7fv25\" (UID: \"4ece8d30-b5c4-4cfa-81b0-49e687d53039\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.209026 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.208847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4ece8d30-b5c4-4cfa-81b0-49e687d53039-podres\") pod \"perf-node-gather-daemonset-7fv25\" (UID: \"4ece8d30-b5c4-4cfa-81b0-49e687d53039\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.209167 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.208863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4ece8d30-b5c4-4cfa-81b0-49e687d53039-proc\") pod \"perf-node-gather-daemonset-7fv25\" (UID: \"4ece8d30-b5c4-4cfa-81b0-49e687d53039\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.209345 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.208951    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ece8d30-b5c4-4cfa-81b0-49e687d53039-lib-modules\") pod \"perf-node-gather-daemonset-7fv25\" (UID: \"4ece8d30-b5c4-4cfa-81b0-49e687d53039\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.209497 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.208951    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4ece8d30-b5c4-4cfa-81b0-49e687d53039-podres\") pod \"perf-node-gather-daemonset-7fv25\" (UID: \"4ece8d30-b5c4-4cfa-81b0-49e687d53039\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.209627 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.209154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h25l\" (UniqueName: \"kubernetes.io/projected/4ece8d30-b5c4-4cfa-81b0-49e687d53039-kube-api-access-9h25l\") pod \"perf-node-gather-daemonset-7fv25\" (UID: \"4ece8d30-b5c4-4cfa-81b0-49e687d53039\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.223864 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.218607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h25l\" (UniqueName: \"kubernetes.io/projected/4ece8d30-b5c4-4cfa-81b0-49e687d53039-kube-api-access-9h25l\") pod \"perf-node-gather-daemonset-7fv25\" (UID: \"4ece8d30-b5c4-4cfa-81b0-49e687d53039\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.307354 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.307265    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.594768 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.594645    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"]
Sep 16 22:01:43.599097 ip-10-0-136-118 kubenswrapper[2563]: W0916 22:01:43.599065    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4ece8d30_b5c4_4cfa_81b0_49e687d53039.slice/crio-7fe6da457483765a4f08c2ea9e323256a7fa21cb4955d678d7a20c01d76c9c39 WatchSource:0}: Error finding container 7fe6da457483765a4f08c2ea9e323256a7fa21cb4955d678d7a20c01d76c9c39: Status 404 returned error can't find the container with id 7fe6da457483765a4f08c2ea9e323256a7fa21cb4955d678d7a20c01d76c9c39
Sep 16 22:01:43.947153 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.946213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25" event={"ID":"4ece8d30-b5c4-4cfa-81b0-49e687d53039","Type":"ContainerStarted","Data":"8fedd106a38e22b347ff3d3bc30a22976e7d8f90f1b50d81ef05c92d87d4fe3d"}
Sep 16 22:01:43.947153 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.946278    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25" event={"ID":"4ece8d30-b5c4-4cfa-81b0-49e687d53039","Type":"ContainerStarted","Data":"7fe6da457483765a4f08c2ea9e323256a7fa21cb4955d678d7a20c01d76c9c39"}
Sep 16 22:01:43.947153 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.947094    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:43.962197 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:43.962119    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25" podStartSLOduration=1.96209599 podStartE2EDuration="1.96209599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 22:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 22:01:43.96097382 +0000 UTC m=+2363.861183826" watchObservedRunningTime="2026-09-16 22:01:43.96209599 +0000 UTC m=+2363.862305996"
Sep 16 22:01:45.243512 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:45.243472    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vlfj5_2dde1c15-7080-4d1e-afc3-6b91a8555c10/dns/0.log"
Sep 16 22:01:45.263289 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:45.263257    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vlfj5_2dde1c15-7080-4d1e-afc3-6b91a8555c10/kube-rbac-proxy/0.log"
Sep 16 22:01:45.287141 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:45.287115    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ftdzk_0f48f3a5-577d-426d-8288-75e7a451c50f/dns-node-resolver/0.log"
Sep 16 22:01:45.868892 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:45.868855    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-765979c47f-j8hjs_b084665b-6abf-4507-8dad-41d4af410da4/registry/0.log"
Sep 16 22:01:45.942909 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:45.942881    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xfzln_fd1d301e-815d-4471-a53a-b7a3bd452da8/node-ca/0.log"
Sep 16 22:01:46.866788 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:46.866760    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd_f26de3a9-5d79-44e6-9fae-ebb703fb3020/istio-proxy/0.log"
Sep 16 22:01:47.669405 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:47.669370    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f65bc968c-wszkq_d679b692-a50b-4df9-9f32-42c34a9b58d5/kube-auth-proxy/0.log"
Sep 16 22:01:47.699395 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:47.699346    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bplw6_8ac674de-6254-4602-8a24-e03a8a075fa9/istio-proxy/0.log"
Sep 16 22:01:47.703246 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:47.703219    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bplw6_8ac674de-6254-4602-8a24-e03a8a075fa9/istio-proxy/1.log"
Sep 16 22:01:48.759890 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:48.759834    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vtbqv_6e8d3c7a-f1d2-4bcf-a3dd-7898c531d61b/serve-healthcheck-canary/0.log"
Sep 16 22:01:49.293176 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:49.293144    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-khxgl_efdae95d-7f63-4795-998a-ed27ce7b8131/kube-rbac-proxy/0.log"
Sep 16 22:01:49.313989 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:49.313961    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-khxgl_efdae95d-7f63-4795-998a-ed27ce7b8131/exporter/0.log"
Sep 16 22:01:49.334261 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:49.334235    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-khxgl_efdae95d-7f63-4795-998a-ed27ce7b8131/extractor/0.log"
Sep 16 22:01:50.973102 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:50.973076    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-7fv25"
Sep 16 22:01:52.203034 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:52.202960    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bvhxd_7b8b352f-26b5-4ec0-a008-c1519d1ab4c1/server/0.log"
Sep 16 22:01:52.323253 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:52.323216    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-bjt9h_f69a6ecb-18e1-4bd7-82ba-d5baf04fd01b/manager/0.log"
Sep 16 22:01:52.538769 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:52.538670    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df9d657c6-tvj4m_0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0/manager/0.log"
Sep 16 22:01:52.564781 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:52.564749    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df9d657c6-tvj4m_0c5765a4-8e84-49c6-8ce0-f9b20bedf8e0/copy-manifests/0.log"
Sep 16 22:01:53.438839 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:53.438803    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd_f26de3a9-5d79-44e6-9fae-ebb703fb3020/istio-proxy/0.log"
Sep 16 22:01:53.846050 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:53.845982    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5dfdd9df77-qht25_9b8ad92e-86a7-4d1f-920f-b7c1117c920e/manager/0.log"
Sep 16 22:01:54.242542 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:54.242514    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f65bc968c-wszkq_d679b692-a50b-4df9-9f32-42c34a9b58d5/kube-auth-proxy/0.log"
Sep 16 22:01:54.267483 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:54.267445    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bplw6_8ac674de-6254-4602-8a24-e03a8a075fa9/istio-proxy/0.log"
Sep 16 22:01:54.270835 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:54.270799    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bplw6_8ac674de-6254-4602-8a24-e03a8a075fa9/istio-proxy/1.log"
Sep 16 22:01:55.780880 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:55.780849    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-xsdpd_f26de3a9-5d79-44e6-9fae-ebb703fb3020/istio-proxy/0.log"
Sep 16 22:01:56.604531 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:56.604421    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f65bc968c-wszkq_d679b692-a50b-4df9-9f32-42c34a9b58d5/kube-auth-proxy/0.log"
Sep 16 22:01:56.634796 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:56.634649    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bplw6_8ac674de-6254-4602-8a24-e03a8a075fa9/istio-proxy/0.log"
Sep 16 22:01:56.638913 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:01:56.638885    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bplw6_8ac674de-6254-4602-8a24-e03a8a075fa9/istio-proxy/1.log"
Sep 16 22:02:00.414533 ip-10-0-136-118 kubenswrapper[2563]: I0916 22:02:00.414499    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7m8bq_a0721a6e-ab7d-4fc9-b5d7-186e30bee273/kube-multus/0.log"