Sep 17 03:17:23.802575 ip-10-0-130-153 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 03:17:23.802586 ip-10-0-130-153 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 03:17:23.802596 ip-10-0-130-153 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 03:17:23.802784 ip-10-0-130-153 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 03:17:34.052295 ip-10-0-130-153 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 03:17:34.052311 ip-10-0-130-153 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e23549907e944f1298c5cd0906167aa2 --
Sep 17 03:19:07.929211 ip-10-0-130-153 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 03:19:08.356542 ip-10-0-130-153 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 03:19:08.356542 ip-10-0-130-153 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 03:19:08.356542 ip-10-0-130-153 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 03:19:08.356542 ip-10-0-130-153 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 03:19:08.356542 ip-10-0-130-153 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 03:19:08.357562 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.357499    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 03:19:08.362279 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362260    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 03:19:08.362279 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362276    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 03:19:08.362279 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362281    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 03:19:08.362279 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362285    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362290    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362293    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362297    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362300    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362303    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362306    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362309    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362311    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362314    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362317    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362319    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362322    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362325    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362327    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362330    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362333    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362335    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362338    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362343    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 03:19:08.362427 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362347    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362350    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362354    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362357    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362361    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362364    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362367    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362370    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362372    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362375    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362378    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362381    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362383    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362386    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362390    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362393    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362396    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362398    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362401    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362404    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 03:19:08.362930 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362406    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362409    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362411    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362414    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362417    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362419    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362422    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362425    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362428    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362431    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362434    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362438    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362443    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362445    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362448    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362451    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362454    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362457    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362459    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 03:19:08.363491 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362462    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362464    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362467    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362470    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362472    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362475    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362478    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362481    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362484    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362487    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362490    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362492    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362497    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362501    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362504    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362507    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362510    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362514    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362517    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362520    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 03:19:08.363976 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362523    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362527    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362529    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362532    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362535    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362537    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362540    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362542    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362546    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362550    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362552    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362555    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362558    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362560    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362563    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362566    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 03:19:08.364463 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.362568    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364477    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364485    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364488    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364492    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364496    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364500    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364504    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364507    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364509    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364512    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364516    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364519    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364522    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364525    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364528    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364530    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364533    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364536    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 03:19:08.364852 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364539    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364542    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364544    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364547    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364549    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364552    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364555    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364557    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364560    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364562    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364566    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364570    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364573    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364576    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364579    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364583    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364586    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364588    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364591    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 03:19:08.365349 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364594    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364597    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364600    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364602    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364605    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364607    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364610    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364612    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364615    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364618    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364620    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364623    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364625    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364628    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364631    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364634    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364636    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364639    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364641    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 03:19:08.365845 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364644    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364646    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364649    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364651    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364654    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364656    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364659    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364661    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364664    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364667    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364670    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364672    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364675    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364678    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364680    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364683    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364685    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364688    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364690    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364693    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 03:19:08.366336 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364695    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364698    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364700    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364703    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364705    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364708    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364710    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364713    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364715    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364718    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364720    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364723    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364727    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364729    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364732    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364735    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364737    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364741    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364743    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364746    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 03:19:08.366833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364748    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364751    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.364754    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365171    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365177    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365181    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365184    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365187    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365190    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365192    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365195    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365198    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365200    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365203    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365206    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365208    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365211    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365213    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365215    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365218    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 03:19:08.367338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365221    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365224    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365227    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365229    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365231    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365234    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365236    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365239    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365241    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365244    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365247    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365249    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365252    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365255    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365257    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365262    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365265    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365269    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365272    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365274    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 03:19:08.367839 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365277    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365280    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365283    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365286    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365288    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365290    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365293    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365295    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365298    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365300    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365302    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365305    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365308    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365310    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365313    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365315    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365318    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365320    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365323    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365325    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 03:19:08.368354 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365327    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365330    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365333    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365336    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365338    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365342    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365345    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365347    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365350    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365353    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365355    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365358    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365360    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365363    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365366    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365368    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365371    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365373    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365376    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 03:19:08.368837 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365378    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365381    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365383    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365386    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365388    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365391    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365393    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365398    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365402    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365405    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365407    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365411    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365414    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365417    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365422    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365425    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365428    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365431    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365434    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 03:19:08.369366 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365436    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365439    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365441    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.365444    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365513    2557 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365547    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365553    2557 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365557    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365561    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365565    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365569    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365573    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365576    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365580    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365583    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365586    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365589    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365593    2557 flags.go:64] FLAG: --cgroup-root=""
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365595    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365598    2557 flags.go:64] FLAG: --client-ca-file=""
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365602    2557 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365605    2557 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365609    2557 flags.go:64] FLAG: --cluster-domain=""
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365612    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 03:19:08.369824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365615    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365618    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365622    2557 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365627    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365630    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365633    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365636    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365639    2557 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365642    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365645    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365649    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365652    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365659    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365662    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365666    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365669    2557 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365671    2557 flags.go:64] FLAG: --enable-server="true"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365674    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365679    2557 flags.go:64] FLAG: --event-burst="100"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365682    2557 flags.go:64] FLAG: --event-qps="50"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365685    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365688    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365692    2557 flags.go:64] FLAG: --eviction-hard=""
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365695    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 03:19:08.370422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365698    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365701    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365704    2557 flags.go:64] FLAG: --eviction-soft=""
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365707    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365710    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365714    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365717    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365719    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365722    2557 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365725    2557 flags.go:64] FLAG: --feature-gates=""
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365729    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365734    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365737    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365740    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365743    2557 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365746    2557 flags.go:64] FLAG: --help="false"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365749    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365752    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365755    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365758    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365761    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365764    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365767    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 03:19:08.371081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365770    2557 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365773    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365776    2557 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365779    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365782    2557 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365785    2557 flags.go:64] FLAG: --kube-reserved=""
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365788    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365790    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365794    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365796    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365799    2557 flags.go:64] FLAG: --lock-file=""
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365802    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365805    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365808    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365813    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365816    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365819    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365822    2557 flags.go:64] FLAG: --logging-format="text"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365824    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365828    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365832    2557 flags.go:64] FLAG: --manifest-url=""
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365835    2557 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365839    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365842    2557 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365846    2557 flags.go:64] FLAG: --max-pods="110"
Sep 17 03:19:08.371663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365849    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365852    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365855    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365858    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365861    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365878    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365881    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365888    2557 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365891    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365894    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365897    2557 flags.go:64] FLAG: --pod-cidr=""
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365900    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365905    2557 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365908    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365911    2557 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365914    2557 flags.go:64] FLAG: --port="10250"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365917    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365920    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05836f1c0fc0a9b1e"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365923    2557 flags.go:64] FLAG: --qos-reserved=""
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365926    2557 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365929    2557 flags.go:64] FLAG: --register-node="true"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365932    2557 flags.go:64] FLAG: --register-with-taints=""
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365936    2557 flags.go:64] FLAG: --registry-burst="10"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365939    2557 flags.go:64] FLAG: --registry-qps="5"
Sep 17 03:19:08.372330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365942    2557 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365945    2557 flags.go:64] FLAG: --reserved-memory=""
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365949    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365952    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365956    2557 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365959    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365963    2557 flags.go:64] FLAG: --runonce="false"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365966    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365969    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365972    2557 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365975    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365978    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365981    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365984    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365987    2557 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365990    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365993    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365996    2557 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.365999    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366002    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366005    2557 flags.go:64] FLAG: --system-cgroups=""
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366007    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366012    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366015    2557 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366018    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 03:19:08.372968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366022    2557 flags.go:64] FLAG: --tls-min-version=""
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366025    2557 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366028    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366031    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366034    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366036    2557 flags.go:64] FLAG: --v="2"
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366040    2557 flags.go:64] FLAG: --version="false"
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366044    2557 flags.go:64] FLAG: --vmodule=""
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366048    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.366051    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366154    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366159    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366163    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366168    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366170    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366173    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366176    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366179    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366181    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366184    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366187    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366191    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 03:19:08.373585 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366195    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366198    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366202    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366205    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366208    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366211    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366214    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366217    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366220    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366223    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366225    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366228    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366231    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366234    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366237    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366240    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366243    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366245    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366248    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 03:19:08.374154 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366251    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366253    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366257    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366260    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366263    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366266    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366268    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366271    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366274    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366277    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366280    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366282    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366284    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366287    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366289    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366292    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366295    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366297    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366300    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366302    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 03:19:08.374655 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366305    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366307    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366310    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366312    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366315    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366318    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366320    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366323    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366325    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366328    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366330    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366333    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366336    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366338    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366342    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366344    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366349    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366352    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366354    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366357    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 03:19:08.375172 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366360    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366362    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366364    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366367    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366370    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366372    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366375    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366377    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366380    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366382    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366385    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366387    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366390    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366392    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366395    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366397    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366408    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366410    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366413    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366416    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 03:19:08.375668 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366419    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 03:19:08.376176 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366421    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 03:19:08.376176 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366424    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 03:19:08.376176 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366428    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 03:19:08.376176 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366432    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 03:19:08.376176 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366435    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 03:19:08.376176 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366438    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 03:19:08.376176 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.366441    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 03:19:08.376176 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.367182    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 03:19:08.376176 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.374027    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 03:19:08.376176 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.374041    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 03:19:08.376176 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.374975    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 03:19:08.376176 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.374983    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 03:19:08.376176 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375037    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 03:19:08.376176 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375042    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375045    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375048    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375051    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375054    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375057    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375060    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375064    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375069    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375072    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375075    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375078    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375081    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375084    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375087    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375089    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375092    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375095    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375098    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375101    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 03:19:08.376528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375103    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375106    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375109    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375111    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375114    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375116    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375119    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375121    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375125    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375127    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375130    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375132    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375135    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375138    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375141    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375143    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375146    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375148    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375151    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375153    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 03:19:08.377040 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375156    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375158    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375163    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375166    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375170    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375172    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375175    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375178    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375180    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375183    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375186    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375189    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375191    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375194    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375196    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375199    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375201    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375204    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375207    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 03:19:08.377528 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375210    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375213    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375216    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375218    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375221    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375224    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375226    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375229    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375231    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375234    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375236    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375239    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375241    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375244    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375247    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375249    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375252    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375254    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375257    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 03:19:08.378046 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375259    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375262    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375264    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375267    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375269    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375272    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375275    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375278    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375282    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375285    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375288    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375292    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375294    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375297    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375301    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375303    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375307    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375309    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375312    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 03:19:08.378517 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375314    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 03:19:08.378992 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.375320    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 03:19:08.378992 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375427    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 03:19:08.378992 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375431    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 03:19:08.378992 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375434    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 03:19:08.378992 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375437    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 03:19:08.378992 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375440    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 03:19:08.378992 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375443    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 03:19:08.378992 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375445    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 03:19:08.378992 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375448    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 03:19:08.378992 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375451    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 03:19:08.378992 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375453    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 03:19:08.378992 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375457    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 03:19:08.378992 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375461    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 03:19:08.378992 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375464    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375467    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375470    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375472    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375475    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375477    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375480    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375484    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375487    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375490    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375493    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375495    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375498    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375501    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375504    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375506    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375509    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375512    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375514    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375517    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 03:19:08.379342 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375520    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375522    2557 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375525    2557 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375527    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375530    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375532    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375535    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375537    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375540    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375542    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375545    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375549    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375552    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375554    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375557    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375559    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375562    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375565    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375567    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375570    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 03:19:08.379836 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375572    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375575    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375577    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375579    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375582    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375585    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375588    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375590    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375593    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375596    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375599    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375601    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375604    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375606    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375609    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375611    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375614    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375616    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375618    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 03:19:08.380423 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375621    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375623    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375626    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375629    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375631    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375633    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375636    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375639    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375641    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375644    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375649    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375651    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375654    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375656    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375659    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375661    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375664    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375667    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375670    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375672    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 03:19:08.380905 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375675    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 03:19:08.381397 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375678    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 03:19:08.381397 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375680    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 03:19:08.381397 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375684    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 03:19:08.381397 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375686    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 03:19:08.381397 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375689    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 03:19:08.381397 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375691    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 03:19:08.381397 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:08.375694    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 03:19:08.381397 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.375699    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 03:19:08.381397 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.375795    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 03:19:08.381397 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.378133    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 03:19:08.381397 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.379157    2557 server.go:1019] "Starting client certificate rotation"
Sep 17 03:19:08.381397 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.379256    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 03:19:08.381397 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.379294    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 03:19:08.382918 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.382898    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 03:19:08.384104 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.384061    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 03:19:08.392440 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.392423    2557 log.go:25] "Validated CRI v1 runtime API"
Sep 17 03:19:08.395676 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.395657    2557 log.go:25] "Validated CRI v1 image API"
Sep 17 03:19:08.396481 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.396463    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 03:19:08.398848 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.398829    2557 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 866379ef-f2bf-405d-a93c-ca00e6ce9896:/dev/nvme0n1p4 e4433114-44dd-4f64-9f3a-cb84913f75e5:/dev/nvme0n1p3]
Sep 17 03:19:08.398936 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.398846    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 03:19:08.404859 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.404759    2557 manager.go:216] Machine: {Timestamp:2026-09-17 03:19:08.403365843 +0000 UTC m=+0.369789194 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3068420 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27907d19e0f0fd3aed59c3fd402dc0 SystemUUID:ec27907d-19e0-f0fd-3aed-59c3fd402dc0 BootID:e2354990-7e94-4f12-98c5-cd0906167aa2 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e4:c9:48:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e4:c9:48:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:80:ae:1b:ce:9d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 03:19:08.404859 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.404854    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 03:19:08.404959 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.404929    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 03:19:08.408544 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.408512    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 03:19:08.408669 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.408549    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-153.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 03:19:08.409413 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.409403    2557 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 03:19:08.409445 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.409415    2557 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 03:19:08.409445 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.409423    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 03:19:08.410280 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.410269    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 03:19:08.411584 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.411573    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 03:19:08.412437 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.412421    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 03:19:08.416164 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.416152    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 03:19:08.416293 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.416255    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 03:19:08.416293 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.416280    2557 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 03:19:08.416293 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.416288    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 03:19:08.416393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.416300    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 03:19:08.416393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.416310    2557 kubelet.go:398] "Adding apiserver pod source"
Sep 17 03:19:08.416393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.416319    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 03:19:08.416670 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.416655    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 03:19:08.416716 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.416684    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 03:19:08.419046 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.419028    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 03:19:08.422176 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.422159    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 03:19:08.422224 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.422192    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 03:19:08.424191 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.424177    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 03:19:08.424248 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.424201    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 03:19:08.424248 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.424211    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 03:19:08.424248 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.424220    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 03:19:08.424248 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.424233    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 03:19:08.424248 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.424244    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 03:19:08.424458 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.424264    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 03:19:08.424458 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.424272    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 03:19:08.424458 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.424281    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 03:19:08.424458 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.424290    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 03:19:08.424458 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.424324    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 03:19:08.424458 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.424333    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 03:19:08.425250 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.425236    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 03:19:08.425283 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.425252    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 03:19:08.428192 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.428164    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 03:19:08.428267 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.428189    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-153.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 03:19:08.428267 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.428216    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-153.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 03:19:08.428946 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.428929    2557 server.go:1268] "Started kubelet"
Sep 17 03:19:08.429030 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.428951    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 03:19:08.429076 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.428986    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 03:19:08.429076 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.429069    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 03:19:08.429617 ip-10-0-130-153 systemd[1]: Started Kubernetes Kubelet.
Sep 17 03:19:08.430551 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.430535    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 03:19:08.435450 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.435434    2557 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 03:19:08.435707 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.435687    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 03:19:08.441475 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.440371    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-153.ec2.internal.18d5fd7ee6c1b7e7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-153.ec2.internal,UID:ip-10-0-130-153.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-153.ec2.internal,},FirstTimestamp:2026-09-17 03:19:08.428802023 +0000 UTC m=+0.395225374,LastTimestamp:2026-09-17 03:19:08.428802023 +0000 UTC m=+0.395225374,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-153.ec2.internal,}"
Sep 17 03:19:08.441733 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.441718    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 03:19:08.442433 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.442407    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 03:19:08.442774 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.442635    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:08.443288 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.443271    2557 factory.go:55] Registering systemd factory
Sep 17 03:19:08.443288 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.443290    2557 factory.go:223] Registration of the systemd container factory successfully
Sep 17 03:19:08.443422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.443295    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 03:19:08.443422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.443295    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 03:19:08.443422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.443315    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 03:19:08.443422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.443378    2557 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 03:19:08.443422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.443385    2557 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 03:19:08.443686 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.443508    2557 factory.go:153] Registering CRI-O factory
Sep 17 03:19:08.443686 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.443524    2557 factory.go:223] Registration of the crio container factory successfully
Sep 17 03:19:08.443686 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.443573    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 03:19:08.443686 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.443591    2557 factory.go:103] Registering Raw factory
Sep 17 03:19:08.443686 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.443604    2557 manager.go:1196] Started watching for new ooms in manager
Sep 17 03:19:08.444017 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.444003    2557 manager.go:349] Starting recovery of all containers
Sep 17 03:19:08.445441 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.445414    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-153.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 03:19:08.445544 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.445523    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 03:19:08.455215 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.455159    2557 manager.go:354] Recovery completed
Sep 17 03:19:08.459320 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.459298    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x5kh2"
Sep 17 03:19:08.460750 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.460738    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 03:19:08.462466 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.462449    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 03:19:08.462531 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.462482    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 03:19:08.462531 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.462496    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientPID"
Sep 17 03:19:08.463024 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.463013    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 03:19:08.463024 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.463025    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 03:19:08.463141 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.463041    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 03:19:08.463141 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.463044    2557 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 03:19:08.463684 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.463667    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 03:19:08.463749 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.463697    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 03:19:08.463749 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.463711    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientPID"
Sep 17 03:19:08.463912 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.463821    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-153.ec2.internal.18d5fd7ee8c3623a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-153.ec2.internal,UID:ip-10-0-130-153.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-153.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-153.ec2.internal,},FirstTimestamp:2026-09-17 03:19:08.462465594 +0000 UTC m=+0.428888947,LastTimestamp:2026-09-17 03:19:08.462465594 +0000 UTC m=+0.428888947,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-153.ec2.internal,}"
Sep 17 03:19:08.465169 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.465154    2557 policy_none.go:49] "None policy: Start"
Sep 17 03:19:08.465169 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.465172    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 03:19:08.465271 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.465181    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 03:19:08.466528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.466513    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x5kh2"
Sep 17 03:19:08.467904 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.467889    2557 policy_none.go:47] "Start"
Sep 17 03:19:08.473478 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.472406    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-153.ec2.internal.18d5fd7ee8c3bbd0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-153.ec2.internal,UID:ip-10-0-130-153.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-153.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-153.ec2.internal,},FirstTimestamp:2026-09-17 03:19:08.462488528 +0000 UTC m=+0.428911881,LastTimestamp:2026-09-17 03:19:08.462488528 +0000 UTC m=+0.428911881,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-153.ec2.internal,}"
Sep 17 03:19:08.475206 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.475152    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-153.ec2.internal.18d5fd7ee8c3f4f1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-153.ec2.internal,UID:ip-10-0-130-153.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-153.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-153.ec2.internal,},FirstTimestamp:2026-09-17 03:19:08.462503153 +0000 UTC m=+0.428926506,LastTimestamp:2026-09-17 03:19:08.462503153 +0000 UTC m=+0.428926506,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-153.ec2.internal,}"
Sep 17 03:19:08.509442 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.509425    2557 manager.go:337] "Starting Device Plugin manager"
Sep 17 03:19:08.524845 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.509464    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 03:19:08.524845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.509477    2557 server.go:87] "Starting device plugin registration server"
Sep 17 03:19:08.524845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.509774    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 03:19:08.524845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.509784    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 03:19:08.524845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.509911    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 03:19:08.524845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.510003    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 03:19:08.524845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.510012    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 03:19:08.524845 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.510375    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 03:19:08.524845 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.510418    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:08.543432 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.543414    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 03:19:08.544571 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.544549    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 03:19:08.544639 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.544606    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 03:19:08.544639 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.544629    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 03:19:08.544735 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.544669    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 03:19:08.546809 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.546785    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 03:19:08.610732 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.610673    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 03:19:08.611342 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.611329    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 03:19:08.611409 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.611354    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 03:19:08.611409 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.611370    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientPID"
Sep 17 03:19:08.611409 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.611395    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.620008 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.619993    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.620093 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.620015    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-153.ec2.internal\": node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:08.643980 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.643961    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:08.644973 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.644957    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal"]
Sep 17 03:19:08.645055 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.645028    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 03:19:08.645712 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.645697    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 03:19:08.645786 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.645724    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 03:19:08.645786 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.645738    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientPID"
Sep 17 03:19:08.648063 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.648049    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 03:19:08.648211 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.648197    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.648264 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.648225    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 03:19:08.648683 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.648670    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 03:19:08.648745 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.648688    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 03:19:08.648745 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.648713    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 03:19:08.648745 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.648725    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientPID"
Sep 17 03:19:08.648852 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.648695    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 03:19:08.648852 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.648778    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientPID"
Sep 17 03:19:08.651283 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.651269    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.651353 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.651293    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 03:19:08.651904 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.651884    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 03:19:08.651993 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.651910    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 03:19:08.651993 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.651922    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientPID"
Sep 17 03:19:08.677312 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.677293    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-153.ec2.internal\" not found" node="ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.681439 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.681426    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-153.ec2.internal\" not found" node="ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.744514 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.744498    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:08.744611 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.744536    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/be359c8ab8f51cc294084eab7fd5bdf6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-153.ec2.internal\" (UID: \"be359c8ab8f51cc294084eab7fd5bdf6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.744611 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.744561    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/efbfb6f947577e237e1ef6c693b522be-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal\" (UID: \"efbfb6f947577e237e1ef6c693b522be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.744611 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.744584    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efbfb6f947577e237e1ef6c693b522be-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal\" (UID: \"efbfb6f947577e237e1ef6c693b522be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.844835 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.844813    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:08.844942 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.844859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/be359c8ab8f51cc294084eab7fd5bdf6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-153.ec2.internal\" (UID: \"be359c8ab8f51cc294084eab7fd5bdf6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.844942 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.844907    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/efbfb6f947577e237e1ef6c693b522be-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal\" (UID: \"efbfb6f947577e237e1ef6c693b522be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.844942 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.844932    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efbfb6f947577e237e1ef6c693b522be-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal\" (UID: \"efbfb6f947577e237e1ef6c693b522be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.845083 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.844977    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/be359c8ab8f51cc294084eab7fd5bdf6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-153.ec2.internal\" (UID: \"be359c8ab8f51cc294084eab7fd5bdf6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.845083 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.845018    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/efbfb6f947577e237e1ef6c693b522be-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal\" (UID: \"efbfb6f947577e237e1ef6c693b522be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.845083 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.844981    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efbfb6f947577e237e1ef6c693b522be-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal\" (UID: \"efbfb6f947577e237e1ef6c693b522be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.945410 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:08.945363    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:08.980518 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.980493    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal"
Sep 17 03:19:08.983792 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:08.983768    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal"
Sep 17 03:19:09.045722 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:09.045695    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:09.146193 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:09.146160    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:10.609871 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:10.609849    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:10.710255 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:10.710228    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:10.742413 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:10.742392    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 03:19:10.742809 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:10.742571    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 03:19:10.810323 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:10.810300    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:10.836555 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:10.836528    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 03:14:09 +0000 UTC" deadline="2028-04-04 14:57:10.913296527 +0000 UTC"
Sep 17 03:19:10.836555 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:10.836553    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13571h38m0.076745936s"
Sep 17 03:19:10.863708 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:10.863669    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 03:19:10.865095 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:10.865077    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 03:19:10.865212 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:10.865107    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 03:19:10.865212 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:10.865116    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeHasSufficientPID"
Sep 17 03:19:10.911221 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:10.911202    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:10.915124 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:10.915110    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 03:19:11.011838 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:11.011807    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:11.024401 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:11.024377    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbfb6f947577e237e1ef6c693b522be.slice/crio-1ff174c1be1ecabb8857578117e1a5657eb81c73841a380b197c11e6a332acf6 WatchSource:0}: Error finding container 1ff174c1be1ecabb8857578117e1a5657eb81c73841a380b197c11e6a332acf6: Status 404 returned error can't find the container with id 1ff174c1be1ecabb8857578117e1a5657eb81c73841a380b197c11e6a332acf6
Sep 17 03:19:11.024664 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:11.024643    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe359c8ab8f51cc294084eab7fd5bdf6.slice/crio-33dd92f0f6056399c9f3cc7bcf849305a9bdd5204d6b871304920afd457519f0 WatchSource:0}: Error finding container 33dd92f0f6056399c9f3cc7bcf849305a9bdd5204d6b871304920afd457519f0: Status 404 returned error can't find the container with id 33dd92f0f6056399c9f3cc7bcf849305a9bdd5204d6b871304920afd457519f0
Sep 17 03:19:11.030757 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.030741    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 03:19:11.111896 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:11.111876    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:11.184169 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.184119    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 03:19:11.212816 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:11.212792    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:11.313462 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:11.313439    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-153.ec2.internal\" not found"
Sep 17 03:19:11.384659 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.384636    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 03:19:11.406203 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.406167    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal"
Sep 17 03:19:11.417911 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.417892    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 03:19:11.419016 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.419000    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal"
Sep 17 03:19:11.426767 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.426749    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 03:19:11.781138 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.781115    2557 apiserver.go:52] "Watching apiserver"
Sep 17 03:19:11.789442 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.789415    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 03:19:11.789904 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.789882    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-n5bhs","openshift-network-operator/iptables-alerter-5s97j","openshift-ovn-kubernetes/ovnkube-node-zkpkk","kube-system/global-pull-secret-syncer-mc2fv","kube-system/konnectivity-agent-gg8lm","kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal","openshift-dns/node-resolver-b77f8","openshift-image-registry/node-ca-f6gqp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal","openshift-multus/multus-additional-cni-plugins-l6v5r","openshift-multus/network-metrics-daemon-5d8wb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf","openshift-cluster-node-tuning-operator/tuned-p8t28","openshift-multus/multus-hz26k"]
Sep 17 03:19:11.792421 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.792372    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:11.792526 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:11.792460    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:11.794656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.794533    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:11.794656 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:11.794620    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:11.796757 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.796737    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b77f8"
Sep 17 03:19:11.798720 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.798698    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 03:19:11.798978 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.798959    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j56rg\""
Sep 17 03:19:11.799160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.799144    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 03:19:11.799363 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.799349    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:11.799438 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:11.799418    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:11.800999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.800979    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gg8lm"
Sep 17 03:19:11.803107 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.803084    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 03:19:11.803107 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.803104    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 03:19:11.803349 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.803333    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5vpjv\""
Sep 17 03:19:11.803423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.803409    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.805840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.805218    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 03:19:11.805840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.805399    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 03:19:11.805840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.805608    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 03:19:11.805840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.805405    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 03:19:11.805840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.805786    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c8mtd\""
Sep 17 03:19:11.805840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.805614    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 03:19:11.806162 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.805977    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 03:19:11.806162 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.806123    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f6gqp"
Sep 17 03:19:11.806407 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.806384    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 03:19:11.807818 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.807801    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 03:19:11.808436 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.807972    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 03:19:11.808436 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.807986    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 03:19:11.808436 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.807999    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.808436 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.808222    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qz5vk\""
Sep 17 03:19:11.809711 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.809693    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 03:19:11.810846 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.810530    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5s97j"
Sep 17 03:19:11.812402 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.812385    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 03:19:11.812614 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.812597    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 03:19:11.812773 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.812696    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ll4qf\""
Sep 17 03:19:11.812968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.812899    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 03:19:11.813022 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.812975    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 03:19:11.813111 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.813091    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6ncgd\""
Sep 17 03:19:11.813404 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.813388    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 03:19:11.813701 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.813681    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 03:19:11.814037 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.814018    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 03:19:11.814269 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.814230    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:11.815048 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815024    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-var-lib-openvswitch\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.815134 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815060    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-cni-netd\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.815134 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815084    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9654da7-e49f-41b1-b764-41002e5bdb28-host\") pod \"node-ca-f6gqp\" (UID: \"c9654da7-e49f-41b1-b764-41002e5bdb28\") " pod="openshift-image-registry/node-ca-f6gqp"
Sep 17 03:19:11.815134 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815106    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8257d8ba-f173-4c72-b229-b9cc0326cf73-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.815318 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815143    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8257d8ba-f173-4c72-b229-b9cc0326cf73-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.815318 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815171    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs\") pod \"network-metrics-daemon-5d8wb\" (UID: \"c7146f5b-e9f5-4469-8161-756c03f69dac\") " pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:11.815318 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815212    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-run-netns\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.815318 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815234    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-run-ovn\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.815318 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815257    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-cni-bin\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.815318 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815275    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:11.815318 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815281    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4tz\" (UniqueName: \"kubernetes.io/projected/c9654da7-e49f-41b1-b764-41002e5bdb28-kube-api-access-6c4tz\") pod \"node-ca-f6gqp\" (UID: \"c9654da7-e49f-41b1-b764-41002e5bdb28\") " pod="openshift-image-registry/node-ca-f6gqp"
Sep 17 03:19:11.815666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815304    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8257d8ba-f173-4c72-b229-b9cc0326cf73-cni-binary-copy\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.815666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815344    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf71bc53-8158-46f9-9ad2-9380beca6a48-hosts-file\") pod \"node-resolver-b77f8\" (UID: \"cf71bc53-8158-46f9-9ad2-9380beca6a48\") " pod="openshift-dns/node-resolver-b77f8"
Sep 17 03:19:11.815666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815366    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:11.815666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815389    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-systemd-units\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.815666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815413    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.815666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815437    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/386c6867-0049-47d3-90ac-e66fdb3ae814-ovnkube-script-lib\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.815666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815458    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9654da7-e49f-41b1-b764-41002e5bdb28-serviceca\") pod \"node-ca-f6gqp\" (UID: \"c9654da7-e49f-41b1-b764-41002e5bdb28\") " pod="openshift-image-registry/node-ca-f6gqp"
Sep 17 03:19:11.815666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815481    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8257d8ba-f173-4c72-b229-b9cc0326cf73-cnibin\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.815666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815506    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-etc-openvswitch\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.815666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815528    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-log-socket\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.815666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815550    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/386c6867-0049-47d3-90ac-e66fdb3ae814-ovn-node-metrics-cert\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.815666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815593    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af354256-5b4f-4a41-8391-b8145ea28913-agent-certs\") pod \"konnectivity-agent-gg8lm\" (UID: \"af354256-5b4f-4a41-8391-b8145ea28913\") " pod="kube-system/konnectivity-agent-gg8lm"
Sep 17 03:19:11.815666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815631    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-kubelet\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.815666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815654    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/386c6867-0049-47d3-90ac-e66fdb3ae814-ovnkube-config\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.816082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815695    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/386c6867-0049-47d3-90ac-e66fdb3ae814-env-overrides\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.816082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815744    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8257d8ba-f173-4c72-b229-b9cc0326cf73-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.816082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815813    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gw9m\" (UniqueName: \"kubernetes.io/projected/c7146f5b-e9f5-4469-8161-756c03f69dac-kube-api-access-2gw9m\") pod \"network-metrics-daemon-5d8wb\" (UID: \"c7146f5b-e9f5-4469-8161-756c03f69dac\") " pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:11.816082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815850    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af354256-5b4f-4a41-8391-b8145ea28913-konnectivity-ca\") pod \"konnectivity-agent-gg8lm\" (UID: \"af354256-5b4f-4a41-8391-b8145ea28913\") " pod="kube-system/konnectivity-agent-gg8lm"
Sep 17 03:19:11.816082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815878    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-run-systemd\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.816082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815905    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-run-openvswitch\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.816082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815943    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-run-ovn-kubernetes\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.816082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815966    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8257d8ba-f173-4c72-b229-b9cc0326cf73-system-cni-dir\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.816082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.815992    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q827k\" (UniqueName: \"kubernetes.io/projected/cf71bc53-8158-46f9-9ad2-9380beca6a48-kube-api-access-q827k\") pod \"node-resolver-b77f8\" (UID: \"cf71bc53-8158-46f9-9ad2-9380beca6a48\") " pod="openshift-dns/node-resolver-b77f8"
Sep 17 03:19:11.816082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.816007    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 03:19:11.816082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.816021    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1b8633f-f1cc-440b-96e6-7e0b526f6583-kubelet-config\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:11.816082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.816043    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1b8633f-f1cc-440b-96e6-7e0b526f6583-dbus\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:11.816082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.816063    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-slash\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.816469 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.816106    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvwz\" (UniqueName: \"kubernetes.io/projected/386c6867-0049-47d3-90ac-e66fdb3ae814-kube-api-access-kkvwz\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.816469 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.816129    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8lrq\" (UniqueName: \"kubernetes.io/projected/8257d8ba-f173-4c72-b229-b9cc0326cf73-kube-api-access-k8lrq\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.816469 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.816150    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-node-log\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.816469 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.816187    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c47g8\" (UniqueName: \"kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8\") pod \"network-check-target-n5bhs\" (UID: \"f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a\") " pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:11.816469 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.816225    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf71bc53-8158-46f9-9ad2-9380beca6a48-tmp-dir\") pod \"node-resolver-b77f8\" (UID: \"cf71bc53-8158-46f9-9ad2-9380beca6a48\") " pod="openshift-dns/node-resolver-b77f8"
Sep 17 03:19:11.816469 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.816313    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 03:19:11.816469 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.816333    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 03:19:11.816469 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.816462    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 03:19:11.816694 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.816517    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fsfpl\""
Sep 17 03:19:11.817153 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.817001    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 03:19:11.817273 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.817217    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7vs6m\""
Sep 17 03:19:11.817870 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.817564    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.817870 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.817760    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 03:19:11.819891 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.819874    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-br6zv\""
Sep 17 03:19:11.820031 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.820018    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 03:19:11.838771 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.838711    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qm9g5"
Sep 17 03:19:11.846071 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.846052    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qm9g5"
Sep 17 03:19:11.907451 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.907425    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 03:19:11.914618 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.914575    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" event={"ID":"efbfb6f947577e237e1ef6c693b522be","Type":"ContainerStarted","Data":"1ff174c1be1ecabb8857578117e1a5657eb81c73841a380b197c11e6a332acf6"}
Sep 17 03:19:11.916262 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916215    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal" event={"ID":"be359c8ab8f51cc294084eab7fd5bdf6","Type":"ContainerStarted","Data":"33dd92f0f6056399c9f3cc7bcf849305a9bdd5204d6b871304920afd457519f0"}
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916484    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-kubelet\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916520    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/386c6867-0049-47d3-90ac-e66fdb3ae814-env-overrides\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916543    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af354256-5b4f-4a41-8391-b8145ea28913-konnectivity-ca\") pod \"konnectivity-agent-gg8lm\" (UID: \"af354256-5b4f-4a41-8391-b8145ea28913\") " pod="kube-system/konnectivity-agent-gg8lm"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916572    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-socket-dir\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916596    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-registration-dir\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916587    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-kubelet\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916620    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-kubernetes\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916643    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-tuned\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916668    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a62682b-18c3-4dfe-982f-93d43396e8e7-cni-binary-copy\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916693    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-run-ovn-kubernetes\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916735    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q827k\" (UniqueName: \"kubernetes.io/projected/cf71bc53-8158-46f9-9ad2-9380beca6a48-kube-api-access-q827k\") pod \"node-resolver-b77f8\" (UID: \"cf71bc53-8158-46f9-9ad2-9380beca6a48\") " pod="openshift-dns/node-resolver-b77f8"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916755    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-run-ovn-kubernetes\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916760    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1b8633f-f1cc-440b-96e6-7e0b526f6583-kubelet-config\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916802    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92478a76-5ee0-4aa4-b6e9-0d1cbd43793a-iptables-alerter-script\") pod \"iptables-alerter-5s97j\" (UID: \"92478a76-5ee0-4aa4-b6e9-0d1cbd43793a\") " pod="openshift-network-operator/iptables-alerter-5s97j"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916814    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c1b8633f-f1cc-440b-96e6-7e0b526f6583-kubelet-config\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916831    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-modprobe-d\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:11.916866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916856    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-multus-cni-dir\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916884    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-slash\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916910    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvwz\" (UniqueName: \"kubernetes.io/projected/386c6867-0049-47d3-90ac-e66fdb3ae814-kube-api-access-kkvwz\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916933    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mb4d\" (UniqueName: \"kubernetes.io/projected/3a62682b-18c3-4dfe-982f-93d43396e8e7-kube-api-access-8mb4d\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916956    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-sysctl-conf\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.916977    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-sys\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917001    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-run-netns\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917023    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-var-lib-openvswitch\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917046    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9654da7-e49f-41b1-b764-41002e5bdb28-host\") pod \"node-ca-f6gqp\" (UID: \"c9654da7-e49f-41b1-b764-41002e5bdb28\") " pod="openshift-image-registry/node-ca-f6gqp"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917068    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917077    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af354256-5b4f-4a41-8391-b8145ea28913-konnectivity-ca\") pod \"konnectivity-agent-gg8lm\" (UID: \"af354256-5b4f-4a41-8391-b8145ea28913\") " pod="kube-system/konnectivity-agent-gg8lm"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917093    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-run-netns\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917137    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-run-ovn\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917169    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4tz\" (UniqueName: \"kubernetes.io/projected/c9654da7-e49f-41b1-b764-41002e5bdb28-kube-api-access-6c4tz\") pod \"node-ca-f6gqp\" (UID: \"c9654da7-e49f-41b1-b764-41002e5bdb28\") " pod="openshift-image-registry/node-ca-f6gqp"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917178    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-slash\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917207    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9654da7-e49f-41b1-b764-41002e5bdb28-host\") pod \"node-ca-f6gqp\" (UID: \"c9654da7-e49f-41b1-b764-41002e5bdb28\") " pod="openshift-image-registry/node-ca-f6gqp"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917218    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-var-lib-openvswitch\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.917743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917239    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-run-ovn\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917247    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8257d8ba-f173-4c72-b229-b9cc0326cf73-cni-binary-copy\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917264    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf71bc53-8158-46f9-9ad2-9380beca6a48-hosts-file\") pod \"node-resolver-b77f8\" (UID: \"cf71bc53-8158-46f9-9ad2-9380beca6a48\") " pod="openshift-dns/node-resolver-b77f8"
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917293    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917313    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917329    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-cnibin\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917330    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-run-netns\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917366    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917389    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9654da7-e49f-41b1-b764-41002e5bdb28-serviceca\") pod \"node-ca-f6gqp\" (UID: \"c9654da7-e49f-41b1-b764-41002e5bdb28\") " pod="openshift-image-registry/node-ca-f6gqp"
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917395    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf71bc53-8158-46f9-9ad2-9380beca6a48-hosts-file\") pod \"node-resolver-b77f8\" (UID: \"cf71bc53-8158-46f9-9ad2-9380beca6a48\") " pod="openshift-dns/node-resolver-b77f8"
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917403    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8257d8ba-f173-4c72-b229-b9cc0326cf73-cnibin\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917418    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-sysconfig\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917450    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8257d8ba-f173-4c72-b229-b9cc0326cf73-cnibin\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917492    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:11.917544    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:11.917599    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret podName:c1b8633f-f1cc-440b-96e6-7e0b526f6583 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:12.41757213 +0000 UTC m=+3.020668111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret") pod "global-pull-secret-syncer-mc2fv" (UID: "c1b8633f-f1cc-440b-96e6-7e0b526f6583") : object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917627    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/386c6867-0049-47d3-90ac-e66fdb3ae814-env-overrides\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.918545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917685    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-run-k8s-cni-cncf-io\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917714    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-etc-openvswitch\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917739    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-log-socket\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917763    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af354256-5b4f-4a41-8391-b8145ea28913-agent-certs\") pod \"konnectivity-agent-gg8lm\" (UID: \"af354256-5b4f-4a41-8391-b8145ea28913\") " pod="kube-system/konnectivity-agent-gg8lm"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917769    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8257d8ba-f173-4c72-b229-b9cc0326cf73-cni-binary-copy\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpsfn\" (UniqueName: \"kubernetes.io/projected/6aeb0577-0ae0-416e-93de-365b23d5f2f0-kube-api-access-tpsfn\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917811    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-systemd\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917831    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-run\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917854    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-lib-modules\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917859    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9654da7-e49f-41b1-b764-41002e5bdb28-serviceca\") pod \"node-ca-f6gqp\" (UID: \"c9654da7-e49f-41b1-b764-41002e5bdb28\") " pod="openshift-image-registry/node-ca-f6gqp"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917870    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-log-socket\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917880    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/386c6867-0049-47d3-90ac-e66fdb3ae814-ovnkube-config\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917831    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-etc-openvswitch\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.917913    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8257d8ba-f173-4c72-b229-b9cc0326cf73-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918023    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gw9m\" (UniqueName: \"kubernetes.io/projected/c7146f5b-e9f5-4469-8161-756c03f69dac-kube-api-access-2gw9m\") pod \"network-metrics-daemon-5d8wb\" (UID: \"c7146f5b-e9f5-4469-8161-756c03f69dac\") " pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918071    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1b8633f-f1cc-440b-96e6-7e0b526f6583-dbus\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918098    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-host\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:11.919351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918133    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhlnj\" (UniqueName: \"kubernetes.io/projected/e86fd36a-17a4-4c70-8460-94a29e14901d-kube-api-access-xhlnj\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918188    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-run-systemd\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918231    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-run-openvswitch\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918255    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8257d8ba-f173-4c72-b229-b9cc0326cf73-system-cni-dir\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918293    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-hostroot\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918318    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-run-multus-certs\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918341    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-etc-kubernetes\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918367    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8lrq\" (UniqueName: \"kubernetes.io/projected/8257d8ba-f173-4c72-b229-b9cc0326cf73-kube-api-access-k8lrq\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918366    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/386c6867-0049-47d3-90ac-e66fdb3ae814-ovnkube-config\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918390    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8257d8ba-f173-4c72-b229-b9cc0326cf73-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918405    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-sysctl-d\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918414    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8257d8ba-f173-4c72-b229-b9cc0326cf73-system-cni-dir\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918428    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918431    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-node-log\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918472    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c47g8\" (UniqueName: \"kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8\") pod \"network-check-target-n5bhs\" (UID: \"f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a\") " pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918471    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c1b8633f-f1cc-440b-96e6-7e0b526f6583-dbus\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918446    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-run-openvswitch\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.920183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918550    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf71bc53-8158-46f9-9ad2-9380beca6a48-tmp-dir\") pod \"node-resolver-b77f8\" (UID: \"cf71bc53-8158-46f9-9ad2-9380beca6a48\") " pod="openshift-dns/node-resolver-b77f8"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918436    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-run-systemd\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918766    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-var-lib-kubelet\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918562    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-node-log\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918821    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-var-lib-cni-multus\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918864    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-cni-netd\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918888    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8257d8ba-f173-4c72-b229-b9cc0326cf73-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918917    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8257d8ba-f173-4c72-b229-b9cc0326cf73-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918924    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-cni-netd\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918942    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs\") pod \"network-metrics-daemon-5d8wb\" (UID: \"c7146f5b-e9f5-4469-8161-756c03f69dac\") " pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918966    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-device-dir\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.918990    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-multus-conf-dir\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919008    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf71bc53-8158-46f9-9ad2-9380beca6a48-tmp-dir\") pod \"node-resolver-b77f8\" (UID: \"cf71bc53-8158-46f9-9ad2-9380beca6a48\") " pod="openshift-dns/node-resolver-b77f8"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919011    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a62682b-18c3-4dfe-982f-93d43396e8e7-multus-daemon-config\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:11.919043    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919083    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8257d8ba-f173-4c72-b229-b9cc0326cf73-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919088    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-cni-bin\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.920999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919119    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbrjd\" (UniqueName: \"kubernetes.io/projected/92478a76-5ee0-4aa4-b6e9-0d1cbd43793a-kube-api-access-fbrjd\") pod \"iptables-alerter-5s97j\" (UID: \"92478a76-5ee0-4aa4-b6e9-0d1cbd43793a\") " pod="openshift-network-operator/iptables-alerter-5s97j"
Sep 17 03:19:11.921839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919131    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-host-cni-bin\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.921839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919238    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-system-cni-dir\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.921839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919269    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-systemd-units\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.921839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919313    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/386c6867-0049-47d3-90ac-e66fdb3ae814-systemd-units\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.921839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919314    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/386c6867-0049-47d3-90ac-e66fdb3ae814-ovnkube-script-lib\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.921839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919348    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-sys-fs\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:11.921839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919372    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e86fd36a-17a4-4c70-8460-94a29e14901d-tmp\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:11.921839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919372    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8257d8ba-f173-4c72-b229-b9cc0326cf73-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.921839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919394    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-multus-socket-dir-parent\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.921839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919424    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-var-lib-cni-bin\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.921839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919457    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/386c6867-0049-47d3-90ac-e66fdb3ae814-ovn-node-metrics-cert\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.921839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919480    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92478a76-5ee0-4aa4-b6e9-0d1cbd43793a-host-slash\") pod \"iptables-alerter-5s97j\" (UID: \"92478a76-5ee0-4aa4-b6e9-0d1cbd43793a\") " pod="openshift-network-operator/iptables-alerter-5s97j"
Sep 17 03:19:11.921839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.919501    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-var-lib-kubelet\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:11.921839 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:11.919653    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs podName:c7146f5b-e9f5-4469-8161-756c03f69dac nodeName:}" failed. No retries permitted until 2026-09-17 03:19:12.419630733 +0000 UTC m=+3.022726730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs") pod "network-metrics-daemon-5d8wb" (UID: "c7146f5b-e9f5-4469-8161-756c03f69dac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:11.921839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.920184    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/386c6867-0049-47d3-90ac-e66fdb3ae814-ovnkube-script-lib\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.922609 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.922589    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/386c6867-0049-47d3-90ac-e66fdb3ae814-ovn-node-metrics-cert\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.923511 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.923488    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af354256-5b4f-4a41-8391-b8145ea28913-agent-certs\") pod \"konnectivity-agent-gg8lm\" (UID: \"af354256-5b4f-4a41-8391-b8145ea28913\") " pod="kube-system/konnectivity-agent-gg8lm"
Sep 17 03:19:11.931702 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.931669    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvwz\" (UniqueName: \"kubernetes.io/projected/386c6867-0049-47d3-90ac-e66fdb3ae814-kube-api-access-kkvwz\") pod \"ovnkube-node-zkpkk\" (UID: \"386c6867-0049-47d3-90ac-e66fdb3ae814\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:11.932669 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:11.932541    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 03:19:11.932669 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:11.932563    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 03:19:11.932669 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:11.932576    2557 projected.go:196] Error preparing data for projected volume kube-api-access-c47g8 for pod openshift-network-diagnostics/network-check-target-n5bhs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:11.932875 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:11.932685    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8 podName:f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a nodeName:}" failed. No retries permitted until 2026-09-17 03:19:12.432669483 +0000 UTC m=+3.035765484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c47g8" (UniqueName: "kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8") pod "network-check-target-n5bhs" (UID: "f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:11.935127 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.935097    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8lrq\" (UniqueName: \"kubernetes.io/projected/8257d8ba-f173-4c72-b229-b9cc0326cf73-kube-api-access-k8lrq\") pod \"multus-additional-cni-plugins-l6v5r\" (UID: \"8257d8ba-f173-4c72-b229-b9cc0326cf73\") " pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:11.935634 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.935609    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gw9m\" (UniqueName: \"kubernetes.io/projected/c7146f5b-e9f5-4469-8161-756c03f69dac-kube-api-access-2gw9m\") pod \"network-metrics-daemon-5d8wb\" (UID: \"c7146f5b-e9f5-4469-8161-756c03f69dac\") " pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:11.936611 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.936592    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4tz\" (UniqueName: \"kubernetes.io/projected/c9654da7-e49f-41b1-b764-41002e5bdb28-kube-api-access-6c4tz\") pod \"node-ca-f6gqp\" (UID: \"c9654da7-e49f-41b1-b764-41002e5bdb28\") " pod="openshift-image-registry/node-ca-f6gqp"
Sep 17 03:19:11.937209 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:11.937179    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q827k\" (UniqueName: \"kubernetes.io/projected/cf71bc53-8158-46f9-9ad2-9380beca6a48-kube-api-access-q827k\") pod \"node-resolver-b77f8\" (UID: \"cf71bc53-8158-46f9-9ad2-9380beca6a48\") " pod="openshift-dns/node-resolver-b77f8"
Sep 17 03:19:12.020248 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020223    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-cnibin\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.020369 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020262    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-sysconfig\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.020369 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020285    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-run-k8s-cni-cncf-io\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.020369 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020309    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpsfn\" (UniqueName: \"kubernetes.io/projected/6aeb0577-0ae0-416e-93de-365b23d5f2f0-kube-api-access-tpsfn\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:12.020369 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020332    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-systemd\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.020369 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020345    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-run\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.020369 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020358    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-lib-modules\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.020369 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020374    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-host\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020387    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlnj\" (UniqueName: \"kubernetes.io/projected/e86fd36a-17a4-4c70-8460-94a29e14901d-kube-api-access-xhlnj\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020401    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-hostroot\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020417    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-run-multus-certs\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020444    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-etc-kubernetes\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020466    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-sysctl-d\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020501    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-var-lib-kubelet\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020524    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-var-lib-cni-multus\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020558    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-device-dir\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020583    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-multus-conf-dir\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020605    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a62682b-18c3-4dfe-982f-93d43396e8e7-multus-daemon-config\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020630    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbrjd\" (UniqueName: \"kubernetes.io/projected/92478a76-5ee0-4aa4-b6e9-0d1cbd43793a-kube-api-access-fbrjd\") pod \"iptables-alerter-5s97j\" (UID: \"92478a76-5ee0-4aa4-b6e9-0d1cbd43793a\") " pod="openshift-network-operator/iptables-alerter-5s97j"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020653    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-system-cni-dir\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020677    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-sys-fs\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020699    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e86fd36a-17a4-4c70-8460-94a29e14901d-tmp\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020721    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-multus-socket-dir-parent\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.020758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020740    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-hostroot\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020791    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-run-multus-certs\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020799    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-var-lib-cni-bin\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020839    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-etc-kubernetes\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020853    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-cnibin\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020900    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-sysconfig\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020935    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-run-k8s-cni-cncf-io\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020937    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-sysctl-d\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020984    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-var-lib-kubelet\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020987    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-systemd\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021074    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-run\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021145    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-lib-modules\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021187    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-host\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021212    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-var-lib-cni-multus\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.020744    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-var-lib-cni-bin\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021252    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92478a76-5ee0-4aa4-b6e9-0d1cbd43793a-host-slash\") pod \"iptables-alerter-5s97j\" (UID: \"92478a76-5ee0-4aa4-b6e9-0d1cbd43793a\") " pod="openshift-network-operator/iptables-alerter-5s97j"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021275    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-var-lib-kubelet\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021303    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-socket-dir\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:12.021528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021326    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-registration-dir\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021349    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-kubernetes\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021373    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-tuned\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021401    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a62682b-18c3-4dfe-982f-93d43396e8e7-cni-binary-copy\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021414    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-device-dir\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021428    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92478a76-5ee0-4aa4-b6e9-0d1cbd43793a-iptables-alerter-script\") pod \"iptables-alerter-5s97j\" (UID: \"92478a76-5ee0-4aa4-b6e9-0d1cbd43793a\") " pod="openshift-network-operator/iptables-alerter-5s97j"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021451    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-modprobe-d\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021454    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-multus-conf-dir\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021475    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-multus-cni-dir\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021501    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mb4d\" (UniqueName: \"kubernetes.io/projected/3a62682b-18c3-4dfe-982f-93d43396e8e7-kube-api-access-8mb4d\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021523    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-sysctl-conf\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021543    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-sys\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021564    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-run-netns\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021591    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021634    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021732    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021772    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92478a76-5ee0-4aa4-b6e9-0d1cbd43793a-host-slash\") pod \"iptables-alerter-5s97j\" (UID: \"92478a76-5ee0-4aa4-b6e9-0d1cbd43793a\") " pod="openshift-network-operator/iptables-alerter-5s97j"
Sep 17 03:19:12.022387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021809    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-var-lib-kubelet\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.023160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021913    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-socket-dir\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:12.023160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021957    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a62682b-18c3-4dfe-982f-93d43396e8e7-multus-daemon-config\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.023160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.021960    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-registration-dir\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:12.023160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.022027    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-kubernetes\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.023160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.022232    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-system-cni-dir\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.023160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.022292    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-sys-fs\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:12.023160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.022343    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-multus-socket-dir-parent\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.023160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.022421    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-modprobe-d\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.023160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.022464    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-sys\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.023160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.022548    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-multus-cni-dir\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.023160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.022572    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-sysctl-conf\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.023160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.022599    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a62682b-18c3-4dfe-982f-93d43396e8e7-host-run-netns\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.023160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.022617    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6aeb0577-0ae0-416e-93de-365b23d5f2f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:12.023160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.022809    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a62682b-18c3-4dfe-982f-93d43396e8e7-cni-binary-copy\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.023160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.022891    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92478a76-5ee0-4aa4-b6e9-0d1cbd43793a-iptables-alerter-script\") pod \"iptables-alerter-5s97j\" (UID: \"92478a76-5ee0-4aa4-b6e9-0d1cbd43793a\") " pod="openshift-network-operator/iptables-alerter-5s97j"
Sep 17 03:19:12.024816 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.024792    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e86fd36a-17a4-4c70-8460-94a29e14901d-tmp\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.024916 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.024897    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e86fd36a-17a4-4c70-8460-94a29e14901d-etc-tuned\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.028873 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.028824    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpsfn\" (UniqueName: \"kubernetes.io/projected/6aeb0577-0ae0-416e-93de-365b23d5f2f0-kube-api-access-tpsfn\") pod \"aws-ebs-csi-driver-node-qxfxf\" (UID: \"6aeb0577-0ae0-416e-93de-365b23d5f2f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:12.029399 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.029375    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlnj\" (UniqueName: \"kubernetes.io/projected/e86fd36a-17a4-4c70-8460-94a29e14901d-kube-api-access-xhlnj\") pod \"tuned-p8t28\" (UID: \"e86fd36a-17a4-4c70-8460-94a29e14901d\") " pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.029490 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.029456    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbrjd\" (UniqueName: \"kubernetes.io/projected/92478a76-5ee0-4aa4-b6e9-0d1cbd43793a-kube-api-access-fbrjd\") pod \"iptables-alerter-5s97j\" (UID: \"92478a76-5ee0-4aa4-b6e9-0d1cbd43793a\") " pod="openshift-network-operator/iptables-alerter-5s97j"
Sep 17 03:19:12.029781 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.029758    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mb4d\" (UniqueName: \"kubernetes.io/projected/3a62682b-18c3-4dfe-982f-93d43396e8e7-kube-api-access-8mb4d\") pod \"multus-hz26k\" (UID: \"3a62682b-18c3-4dfe-982f-93d43396e8e7\") " pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.109809 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.109758    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b77f8"
Sep 17 03:19:12.118180 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.118161    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gg8lm"
Sep 17 03:19:12.122615 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:12.122589    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf71bc53_8158_46f9_9ad2_9380beca6a48.slice/crio-61d6f1fabf86dc591cdfe9ad6bd8c12f27de9f5c4cce636e8451408cbb5c1e37 WatchSource:0}: Error finding container 61d6f1fabf86dc591cdfe9ad6bd8c12f27de9f5c4cce636e8451408cbb5c1e37: Status 404 returned error can't find the container with id 61d6f1fabf86dc591cdfe9ad6bd8c12f27de9f5c4cce636e8451408cbb5c1e37
Sep 17 03:19:12.130113 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.130091    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:12.134887 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.134871    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f6gqp"
Sep 17 03:19:12.142424 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.142404    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l6v5r"
Sep 17 03:19:12.151270 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.151252    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5s97j"
Sep 17 03:19:12.158924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.158903    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf"
Sep 17 03:19:12.171805 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.171780    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p8t28"
Sep 17 03:19:12.179492 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.179469    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hz26k"
Sep 17 03:19:12.303603 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.303580    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 03:19:12.425689 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.425621    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs\") pod \"network-metrics-daemon-5d8wb\" (UID: \"c7146f5b-e9f5-4469-8161-756c03f69dac\") " pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:12.425812 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.425685    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:12.425812 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:12.425800    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:12.425890 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:12.425863    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs podName:c7146f5b-e9f5-4469-8161-756c03f69dac nodeName:}" failed. No retries permitted until 2026-09-17 03:19:13.425844133 +0000 UTC m=+4.028940140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs") pod "network-metrics-daemon-5d8wb" (UID: "c7146f5b-e9f5-4469-8161-756c03f69dac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:12.425952 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:12.425800    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:12.425952 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:12.425945    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret podName:c1b8633f-f1cc-440b-96e6-7e0b526f6583 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:13.425928399 +0000 UTC m=+4.029024399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret") pod "global-pull-secret-syncer-mc2fv" (UID: "c1b8633f-f1cc-440b-96e6-7e0b526f6583") : object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:12.526432 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.526403    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c47g8\" (UniqueName: \"kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8\") pod \"network-check-target-n5bhs\" (UID: \"f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a\") " pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:12.526615 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:12.526597    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 03:19:12.526671 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:12.526620    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 03:19:12.526671 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:12.526630    2557 projected.go:196] Error preparing data for projected volume kube-api-access-c47g8 for pod openshift-network-diagnostics/network-check-target-n5bhs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:12.526767 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:12.526677    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8 podName:f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a nodeName:}" failed. No retries permitted until 2026-09-17 03:19:13.526662884 +0000 UTC m=+4.129758866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c47g8" (UniqueName: "kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8") pod "network-check-target-n5bhs" (UID: "f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:12.849458 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.849359    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 03:14:11 +0000 UTC" deadline="2028-05-13 22:03:08.294951453 +0000 UTC"
Sep 17 03:19:12.849458 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.849402    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14514h43m55.445552806s"
Sep 17 03:19:12.919270 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:12.919228    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b77f8" event={"ID":"cf71bc53-8158-46f9-9ad2-9380beca6a48","Type":"ContainerStarted","Data":"61d6f1fabf86dc591cdfe9ad6bd8c12f27de9f5c4cce636e8451408cbb5c1e37"}
Sep 17 03:19:13.155294 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:13.155261    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf354256_5b4f_4a41_8391_b8145ea28913.slice/crio-f6742452071cc6d0b31ce21e89ce7d2f541292c919699efe0b7d3780d4bc8127 WatchSource:0}: Error finding container f6742452071cc6d0b31ce21e89ce7d2f541292c919699efe0b7d3780d4bc8127: Status 404 returned error can't find the container with id f6742452071cc6d0b31ce21e89ce7d2f541292c919699efe0b7d3780d4bc8127
Sep 17 03:19:13.176761 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:13.176731    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386c6867_0049_47d3_90ac_e66fdb3ae814.slice/crio-43e80b45585de3298c34b167f1dcd0c2a15841e544e30d67ef7d1565fff38b69 WatchSource:0}: Error finding container 43e80b45585de3298c34b167f1dcd0c2a15841e544e30d67ef7d1565fff38b69: Status 404 returned error can't find the container with id 43e80b45585de3298c34b167f1dcd0c2a15841e544e30d67ef7d1565fff38b69
Sep 17 03:19:13.178867 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:13.178622    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9654da7_e49f_41b1_b764_41002e5bdb28.slice/crio-995857ca04ce2998181798c19fd53cae5175499f0bef354242ab171ecd8e968c WatchSource:0}: Error finding container 995857ca04ce2998181798c19fd53cae5175499f0bef354242ab171ecd8e968c: Status 404 returned error can't find the container with id 995857ca04ce2998181798c19fd53cae5175499f0bef354242ab171ecd8e968c
Sep 17 03:19:13.178964 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:13.178947    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92478a76_5ee0_4aa4_b6e9_0d1cbd43793a.slice/crio-c7b85ee77a02f4aecfc50efbde6fed7be86aad45f93147bada15777863369a8a WatchSource:0}: Error finding container c7b85ee77a02f4aecfc50efbde6fed7be86aad45f93147bada15777863369a8a: Status 404 returned error can't find the container with id c7b85ee77a02f4aecfc50efbde6fed7be86aad45f93147bada15777863369a8a
Sep 17 03:19:13.179762 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:13.179740    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aeb0577_0ae0_416e_93de_365b23d5f2f0.slice/crio-05fa9061126fdd4d9217216cee09e0d01af9979a19a9bb6b2c021967d34a078b WatchSource:0}: Error finding container 05fa9061126fdd4d9217216cee09e0d01af9979a19a9bb6b2c021967d34a078b: Status 404 returned error can't find the container with id 05fa9061126fdd4d9217216cee09e0d01af9979a19a9bb6b2c021967d34a078b
Sep 17 03:19:13.182554 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:13.182486    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a62682b_18c3_4dfe_982f_93d43396e8e7.slice/crio-1d9f083ebc3a2d71283dd3a7e2e802bbe408ff7e2f30844c9dcee525bfdececb WatchSource:0}: Error finding container 1d9f083ebc3a2d71283dd3a7e2e802bbe408ff7e2f30844c9dcee525bfdececb: Status 404 returned error can't find the container with id 1d9f083ebc3a2d71283dd3a7e2e802bbe408ff7e2f30844c9dcee525bfdececb
Sep 17 03:19:13.183101 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:13.183068    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8257d8ba_f173_4c72_b229_b9cc0326cf73.slice/crio-abff3f632950527c9d31abeb69122dd5d6c10e8e26691c01b774506599c00e45 WatchSource:0}: Error finding container abff3f632950527c9d31abeb69122dd5d6c10e8e26691c01b774506599c00e45: Status 404 returned error can't find the container with id abff3f632950527c9d31abeb69122dd5d6c10e8e26691c01b774506599c00e45
Sep 17 03:19:13.434037 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.433987    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:13.434177 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.434144    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs\") pod \"network-metrics-daemon-5d8wb\" (UID: \"c7146f5b-e9f5-4469-8161-756c03f69dac\") " pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:13.434177 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:13.434152    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:13.434298 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:13.434232    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret podName:c1b8633f-f1cc-440b-96e6-7e0b526f6583 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:15.434212891 +0000 UTC m=+6.037308877 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret") pod "global-pull-secret-syncer-mc2fv" (UID: "c1b8633f-f1cc-440b-96e6-7e0b526f6583") : object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:13.434298 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:13.434279    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:13.434402 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:13.434318    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs podName:c7146f5b-e9f5-4469-8161-756c03f69dac nodeName:}" failed. No retries permitted until 2026-09-17 03:19:15.434305728 +0000 UTC m=+6.037401728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs") pod "network-metrics-daemon-5d8wb" (UID: "c7146f5b-e9f5-4469-8161-756c03f69dac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:13.534635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.534605    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c47g8\" (UniqueName: \"kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8\") pod \"network-check-target-n5bhs\" (UID: \"f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a\") " pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:13.534798 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:13.534777    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 03:19:13.534860 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:13.534804    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 03:19:13.534860 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:13.534819    2557 projected.go:196] Error preparing data for projected volume kube-api-access-c47g8 for pod openshift-network-diagnostics/network-check-target-n5bhs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:13.534942 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:13.534868    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8 podName:f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a nodeName:}" failed. No retries permitted until 2026-09-17 03:19:15.53485137 +0000 UTC m=+6.137947375 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-c47g8" (UniqueName: "kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8") pod "network-check-target-n5bhs" (UID: "f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:13.850465 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.850332    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 03:14:11 +0000 UTC" deadline="2028-03-04 21:11:58.429057341 +0000 UTC"
Sep 17 03:19:13.850465 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.850386    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12833h52m44.578675653s"
Sep 17 03:19:13.910138 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.908868    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:13.910138 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:13.908992    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:13.910138 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.909289    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:13.910138 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:13.909389    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:13.910138 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.909657    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:13.910138 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:13.909766    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:13.930956 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.930924    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gg8lm" event={"ID":"af354256-5b4f-4a41-8391-b8145ea28913","Type":"ContainerStarted","Data":"f6742452071cc6d0b31ce21e89ce7d2f541292c919699efe0b7d3780d4bc8127"}
Sep 17 03:19:13.944476 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.944449    2557 generic.go:356] "Generic (PLEG): container finished" podID="efbfb6f947577e237e1ef6c693b522be" containerID="01abc54b99ea101f14fe6e26b0b8509b783a1aba18c13328bcbee5926d4f1903" exitCode=0
Sep 17 03:19:13.944995 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.944972    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" event={"ID":"efbfb6f947577e237e1ef6c693b522be","Type":"ContainerDied","Data":"01abc54b99ea101f14fe6e26b0b8509b783a1aba18c13328bcbee5926d4f1903"}
Sep 17 03:19:13.950145 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.950122    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hz26k" event={"ID":"3a62682b-18c3-4dfe-982f-93d43396e8e7","Type":"ContainerStarted","Data":"1d9f083ebc3a2d71283dd3a7e2e802bbe408ff7e2f30844c9dcee525bfdececb"}
Sep 17 03:19:13.953929 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.953897    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f6gqp" event={"ID":"c9654da7-e49f-41b1-b764-41002e5bdb28","Type":"ContainerStarted","Data":"995857ca04ce2998181798c19fd53cae5175499f0bef354242ab171ecd8e968c"}
Sep 17 03:19:13.957369 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.957346    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal" event={"ID":"be359c8ab8f51cc294084eab7fd5bdf6","Type":"ContainerStarted","Data":"6447ab69d6fcd22418abc3cb1d40c4a81eb69de28ababcbb81c88b9c59b5dfa7"}
Sep 17 03:19:13.959345 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.959323    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p8t28" event={"ID":"e86fd36a-17a4-4c70-8460-94a29e14901d","Type":"ContainerStarted","Data":"4d61b1db541db131cb8573cb4802a59fe3072ca6201a1e44caf9d612cffc9091"}
Sep 17 03:19:13.963878 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.963855    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6v5r" event={"ID":"8257d8ba-f173-4c72-b229-b9cc0326cf73","Type":"ContainerStarted","Data":"abff3f632950527c9d31abeb69122dd5d6c10e8e26691c01b774506599c00e45"}
Sep 17 03:19:13.966711 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.966689    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf" event={"ID":"6aeb0577-0ae0-416e-93de-365b23d5f2f0","Type":"ContainerStarted","Data":"05fa9061126fdd4d9217216cee09e0d01af9979a19a9bb6b2c021967d34a078b"}
Sep 17 03:19:13.968261 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.968237    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5s97j" event={"ID":"92478a76-5ee0-4aa4-b6e9-0d1cbd43793a","Type":"ContainerStarted","Data":"c7b85ee77a02f4aecfc50efbde6fed7be86aad45f93147bada15777863369a8a"}
Sep 17 03:19:13.971105 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.971081    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk" event={"ID":"386c6867-0049-47d3-90ac-e66fdb3ae814","Type":"ContainerStarted","Data":"43e80b45585de3298c34b167f1dcd0c2a15841e544e30d67ef7d1565fff38b69"}
Sep 17 03:19:13.972933 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:13.972886    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-153.ec2.internal" podStartSLOduration=2.972869868 podStartE2EDuration="2.972869868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:13.972729375 +0000 UTC m=+4.575825379" watchObservedRunningTime="2026-09-17 03:19:13.972869868 +0000 UTC m=+4.575965873"
Sep 17 03:19:15.449434 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:15.449397    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs\") pod \"network-metrics-daemon-5d8wb\" (UID: \"c7146f5b-e9f5-4469-8161-756c03f69dac\") " pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:15.449884 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:15.449465    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:15.449884 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:15.449599    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:15.449884 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:15.449657    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret podName:c1b8633f-f1cc-440b-96e6-7e0b526f6583 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:19.449638874 +0000 UTC m=+10.052734860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret") pod "global-pull-secret-syncer-mc2fv" (UID: "c1b8633f-f1cc-440b-96e6-7e0b526f6583") : object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:15.450089 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:15.450057    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:15.450142 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:15.450110    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs podName:c7146f5b-e9f5-4469-8161-756c03f69dac nodeName:}" failed. No retries permitted until 2026-09-17 03:19:19.450094349 +0000 UTC m=+10.053190334 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs") pod "network-metrics-daemon-5d8wb" (UID: "c7146f5b-e9f5-4469-8161-756c03f69dac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:15.550709 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:15.550674    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c47g8\" (UniqueName: \"kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8\") pod \"network-check-target-n5bhs\" (UID: \"f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a\") " pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:15.550901 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:15.550882    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 03:19:15.550975 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:15.550904    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 03:19:15.550975 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:15.550917    2557 projected.go:196] Error preparing data for projected volume kube-api-access-c47g8 for pod openshift-network-diagnostics/network-check-target-n5bhs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:15.550975 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:15.550966    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8 podName:f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a nodeName:}" failed. No retries permitted until 2026-09-17 03:19:19.55095087 +0000 UTC m=+10.154046868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-c47g8" (UniqueName: "kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8") pod "network-check-target-n5bhs" (UID: "f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:15.910774 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:15.908730    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:15.910774 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:15.908733    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:15.910774 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:15.908857    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:15.910774 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:15.908733    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:15.910774 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:15.908946    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:15.910774 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:15.909008    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:17.910388 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:17.909026    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:17.910388 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:17.909182    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:17.910388 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:17.909495    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:17.910388 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:17.909559    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:17.910388 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:17.909675    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:17.910388 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:17.909727    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:19.492941 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:19.492904    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs\") pod \"network-metrics-daemon-5d8wb\" (UID: \"c7146f5b-e9f5-4469-8161-756c03f69dac\") " pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:19.493506 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:19.492970    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:19.493506 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:19.493068    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:19.493506 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:19.493084    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:19.493506 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:19.493139    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs podName:c7146f5b-e9f5-4469-8161-756c03f69dac nodeName:}" failed. No retries permitted until 2026-09-17 03:19:27.493119963 +0000 UTC m=+18.096215961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs") pod "network-metrics-daemon-5d8wb" (UID: "c7146f5b-e9f5-4469-8161-756c03f69dac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:19.493506 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:19.493158    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret podName:c1b8633f-f1cc-440b-96e6-7e0b526f6583 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:27.493149731 +0000 UTC m=+18.096245712 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret") pod "global-pull-secret-syncer-mc2fv" (UID: "c1b8633f-f1cc-440b-96e6-7e0b526f6583") : object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:19.593331 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:19.593243    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c47g8\" (UniqueName: \"kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8\") pod \"network-check-target-n5bhs\" (UID: \"f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a\") " pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:19.593471 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:19.593443    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 03:19:19.593471 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:19.593475    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 03:19:19.593619 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:19.593488    2557 projected.go:196] Error preparing data for projected volume kube-api-access-c47g8 for pod openshift-network-diagnostics/network-check-target-n5bhs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:19.593619 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:19.593549    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8 podName:f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a nodeName:}" failed. No retries permitted until 2026-09-17 03:19:27.593529971 +0000 UTC m=+18.196625964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-c47g8" (UniqueName: "kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8") pod "network-check-target-n5bhs" (UID: "f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:19.909836 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:19.909714    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:19.909836 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:19.909814    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:19.910433 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:19.910120    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:19.910433 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:19.910246    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:19.910433 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:19.910294    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:19.910433 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:19.910365    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:21.908815 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:21.908784    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:21.909234 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:21.908784    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:21.909234 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:21.908891    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:21.909234 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:21.908986    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:21.909234 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:21.908784    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:21.909234 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:21.909062    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:23.909128 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:23.909094    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:23.909550 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:23.909227    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:23.909550 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:23.909327    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:23.909550 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:23.909345    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:23.909550 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:23.909449    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:23.909767 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:23.909685    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:25.908526 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:25.908494    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:25.908526 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:25.908517    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:25.909026 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:25.908605    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:25.909026 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:25.908644    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:25.909026 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:25.908647    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:25.909026 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:25.908698    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:27.553827 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:27.553773    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs\") pod \"network-metrics-daemon-5d8wb\" (UID: \"c7146f5b-e9f5-4469-8161-756c03f69dac\") " pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:27.554324 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:27.553938    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:27.554324 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:27.553962    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:27.554324 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:27.554019    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs podName:c7146f5b-e9f5-4469-8161-756c03f69dac nodeName:}" failed. No retries permitted until 2026-09-17 03:19:43.553998342 +0000 UTC m=+34.157094339 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs") pod "network-metrics-daemon-5d8wb" (UID: "c7146f5b-e9f5-4469-8161-756c03f69dac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:27.554324 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:27.554134    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:27.554324 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:27.554210    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret podName:c1b8633f-f1cc-440b-96e6-7e0b526f6583 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:43.554178049 +0000 UTC m=+34.157274043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret") pod "global-pull-secret-syncer-mc2fv" (UID: "c1b8633f-f1cc-440b-96e6-7e0b526f6583") : object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:27.655180 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:27.655155    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c47g8\" (UniqueName: \"kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8\") pod \"network-check-target-n5bhs\" (UID: \"f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a\") " pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:27.655333 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:27.655317    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 03:19:27.655397 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:27.655338    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 03:19:27.655397 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:27.655348    2557 projected.go:196] Error preparing data for projected volume kube-api-access-c47g8 for pod openshift-network-diagnostics/network-check-target-n5bhs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:27.655397 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:27.655393    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8 podName:f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a nodeName:}" failed. No retries permitted until 2026-09-17 03:19:43.655377149 +0000 UTC m=+34.258473146 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-c47g8" (UniqueName: "kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8") pod "network-check-target-n5bhs" (UID: "f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:27.908381 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:27.908272    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:27.908381 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:27.908284    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:27.908381 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:27.908287    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:27.908623 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:27.908389    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:27.908623 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:27.908466    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:27.908623 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:27.908587    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:29.911997 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:29.911965    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:29.911997 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:29.911985    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:29.912520 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:29.911965    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:29.912520 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:29.912063    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:29.912520 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:29.912150    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:29.912520 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:29.912260    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:31.007234 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.006992    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf" event={"ID":"6aeb0577-0ae0-416e-93de-365b23d5f2f0","Type":"ContainerStarted","Data":"37f30a01fa8d5bf26137e1a8485c21d77bb4343189d207a64230d7ce245bc13d"}
Sep 17 03:19:31.010083 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.010060    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk" event={"ID":"386c6867-0049-47d3-90ac-e66fdb3ae814","Type":"ContainerStarted","Data":"8f59ad8f507cd49cf30f9d5092cf185afef8888e0705632262a4fa3154cbea58"}
Sep 17 03:19:31.010259 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.010122    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk" event={"ID":"386c6867-0049-47d3-90ac-e66fdb3ae814","Type":"ContainerStarted","Data":"ab703387390277e7d621f82638516bafe2da20b03c311814a76cb8ebf55b1ef2"}
Sep 17 03:19:31.010259 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.010153    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk" event={"ID":"386c6867-0049-47d3-90ac-e66fdb3ae814","Type":"ContainerStarted","Data":"d522cfea837aa80d518ca4a4ad90564accd5fc75975347ae41e671054a330d76"}
Sep 17 03:19:31.010259 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.010181    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk" event={"ID":"386c6867-0049-47d3-90ac-e66fdb3ae814","Type":"ContainerStarted","Data":"dc411f6d3783113ee7b41832ed4ed906b662c6fbc9f6017a203ccaf13e38f6db"}
Sep 17 03:19:31.010259 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.010229    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk" event={"ID":"386c6867-0049-47d3-90ac-e66fdb3ae814","Type":"ContainerStarted","Data":"3792e0358f88c669f6ec58e61ec6ee2aeda428ca2686bfe15d4749622b955cb6"}
Sep 17 03:19:31.010424 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.010288    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk" event={"ID":"386c6867-0049-47d3-90ac-e66fdb3ae814","Type":"ContainerStarted","Data":"01fd8ffc3b6dd161be4ceabf0abc08057cba0a09450f2f42e80257746623fb99"}
Sep 17 03:19:31.011467 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.011442    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gg8lm" event={"ID":"af354256-5b4f-4a41-8391-b8145ea28913","Type":"ContainerStarted","Data":"4906671224af9de57afcceedb24dfae9837851bc826d782cbff9a8b8267e5043"}
Sep 17 03:19:31.013688 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.013668    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" event={"ID":"efbfb6f947577e237e1ef6c693b522be","Type":"ContainerStarted","Data":"c20fd30f1e0aa2f3b4a5f53bd9639125601fde3a374bbb72a365116dd4e3d887"}
Sep 17 03:19:31.015181 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.015156    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hz26k" event={"ID":"3a62682b-18c3-4dfe-982f-93d43396e8e7","Type":"ContainerStarted","Data":"c1e49164f9e196e3012d836fb28aaf00180f36c73afc195b89aaff596fe93571"}
Sep 17 03:19:31.016490 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.016471    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f6gqp" event={"ID":"c9654da7-e49f-41b1-b764-41002e5bdb28","Type":"ContainerStarted","Data":"b72a2b411a71bf8f1d57d72eac6f15c7d3e4f593a7b1e4b323b8b1dab0e948ab"}
Sep 17 03:19:31.017816 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.017794    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b77f8" event={"ID":"cf71bc53-8158-46f9-9ad2-9380beca6a48","Type":"ContainerStarted","Data":"fedb180e0790b0dcb9c6849916210cdfaf1ea13a31955f554a22611f8661bc3c"}
Sep 17 03:19:31.019091 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.019073    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p8t28" event={"ID":"e86fd36a-17a4-4c70-8460-94a29e14901d","Type":"ContainerStarted","Data":"93b0c141ea6c1c9e6e472097536c9f985574c6fd18545b4c8512ec05ba8f035c"}
Sep 17 03:19:31.020499 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.020480    2557 generic.go:356] "Generic (PLEG): container finished" podID="8257d8ba-f173-4c72-b229-b9cc0326cf73" containerID="24c72b10157958b667a57db0459253203c18c9876df5e7346645ec144759fae8" exitCode=0
Sep 17 03:19:31.020576 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.020507    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6v5r" event={"ID":"8257d8ba-f173-4c72-b229-b9cc0326cf73","Type":"ContainerDied","Data":"24c72b10157958b667a57db0459253203c18c9876df5e7346645ec144759fae8"}
Sep 17 03:19:31.038005 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.037961    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gg8lm" podStartSLOduration=4.137697207 podStartE2EDuration="21.037951124s" totalImagesPullingTime="16.900253917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:31.024855481 +0000 UTC m=+21.627951495" watchObservedRunningTime="2026-09-17 03:19:31.037951124 +0000 UTC m=+21.641047173"
Sep 17 03:19:31.038097 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.038072    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-b77f8" podStartSLOduration=6.940634524 podStartE2EDuration="21.03806418s" totalImagesPullingTime="14.097429656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:31.037806838 +0000 UTC m=+21.640902842" watchObservedRunningTime="2026-09-17 03:19:31.03806418 +0000 UTC m=+21.641160185"
Sep 17 03:19:31.063666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.063634    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f6gqp" podStartSLOduration=4.183325266 podStartE2EDuration="21.063624177s" totalImagesPullingTime="16.880298911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:31.063556247 +0000 UTC m=+21.666652251" watchObservedRunningTime="2026-09-17 03:19:31.063624177 +0000 UTC m=+21.666720181"
Sep 17 03:19:31.063882 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.063864    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hz26k" podStartSLOduration=3.817495637 podStartE2EDuration="21.063858054s" totalImagesPullingTime="17.246362417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:31.052042129 +0000 UTC m=+21.655138143" watchObservedRunningTime="2026-09-17 03:19:31.063858054 +0000 UTC m=+21.666954057"
Sep 17 03:19:31.095868 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.095835    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p8t28" podStartSLOduration=3.931381272 podStartE2EDuration="21.095825938s" totalImagesPullingTime="17.164444666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:31.095765498 +0000 UTC m=+21.698861501" watchObservedRunningTime="2026-09-17 03:19:31.095825938 +0000 UTC m=+21.698921942"
Sep 17 03:19:31.110479 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.110444    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-153.ec2.internal" podStartSLOduration=20.110434998 podStartE2EDuration="20.110434998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:31.110089063 +0000 UTC m=+21.713185066" watchObservedRunningTime="2026-09-17 03:19:31.110434998 +0000 UTC m=+21.713531000"
Sep 17 03:19:31.741503 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.741372    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 03:19:31.884207 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.884062    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T03:19:31.741500714Z","UUID":"42182c1c-12b2-4563-ab7a-4e147771a733","Handler":null,"Name":"","Endpoint":""}
Sep 17 03:19:31.885606 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.885583    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 03:19:31.885714 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.885614    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 03:19:31.908950 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.908928    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:31.909067 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.908929    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:31.909067 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:31.909055    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:31.909067 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:31.909051    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:31.909253 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:31.909151    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:31.909253 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:31.909238    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:32.024636 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:32.024613    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5s97j" event={"ID":"92478a76-5ee0-4aa4-b6e9-0d1cbd43793a","Type":"ContainerStarted","Data":"510516e2dbacd4cbd40973a08c644adbb982934a6213be8f1ce2ba86f667c897"}
Sep 17 03:19:32.026583 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:32.026501    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf" event={"ID":"6aeb0577-0ae0-416e-93de-365b23d5f2f0","Type":"ContainerStarted","Data":"eabe8e14d4f55d29d0b2738b25a60c4bd40a6c9f377196de0191310f06cbce7a"}
Sep 17 03:19:32.037438 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:32.037401    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5s97j" podStartSLOduration=8.996797389 podStartE2EDuration="22.037387033s" totalImagesPullingTime="13.040589644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:32.036845706 +0000 UTC m=+22.639941711" watchObservedRunningTime="2026-09-17 03:19:32.037387033 +0000 UTC m=+22.640483036"
Sep 17 03:19:33.031437 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:33.031393    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf" event={"ID":"6aeb0577-0ae0-416e-93de-365b23d5f2f0","Type":"ContainerStarted","Data":"fde9c1f2821208c6799c6e04be8e939a30bb51a0bdec46ac71e76aad9413e7cd"}
Sep 17 03:19:33.035827 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:33.035789    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk" event={"ID":"386c6867-0049-47d3-90ac-e66fdb3ae814","Type":"ContainerStarted","Data":"ddc12c1216905597c531d2389049857837156adf201654175238252532fa895c"}
Sep 17 03:19:33.050854 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:33.050783    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qxfxf" podStartSLOduration=4.059634272 podStartE2EDuration="23.050766376s" totalImagesPullingTime="18.991132104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:33.050739171 +0000 UTC m=+23.653835175" watchObservedRunningTime="2026-09-17 03:19:33.050766376 +0000 UTC m=+23.653862374"
Sep 17 03:19:33.908731 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:33.908702    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:33.908962 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:33.908830    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:33.908962 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:33.908866    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:33.908962 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:33.908922    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:33.908962 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:33.908831    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:33.909116 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:33.908986    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:35.908504 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:35.908470    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:35.908504 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:35.908506    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:35.908992 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:35.908597    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:35.908992 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:35.908680    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:35.908992 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:35.908706    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:35.908992 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:35.908788    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:35.912308 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:35.912284    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gg8lm"
Sep 17 03:19:35.912455 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:35.912368    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gg8lm"
Sep 17 03:19:35.912724 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:35.912681    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gg8lm"
Sep 17 03:19:35.913134 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:35.913110    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gg8lm"
Sep 17 03:19:37.048148 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:37.047866    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk" event={"ID":"386c6867-0049-47d3-90ac-e66fdb3ae814","Type":"ContainerStarted","Data":"09b894d850b0a92be23f9db96aea155b0b74c13259fad76844c75b55a76ff26a"}
Sep 17 03:19:37.048992 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:37.048264    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:37.050022 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:37.049994    2557 generic.go:356] "Generic (PLEG): container finished" podID="8257d8ba-f173-4c72-b229-b9cc0326cf73" containerID="5d08008d83bca220f0403dc80e7556b597d59221fbe3d7d98fde7ced0585f8e3" exitCode=0
Sep 17 03:19:37.050154 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:37.050076    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6v5r" event={"ID":"8257d8ba-f173-4c72-b229-b9cc0326cf73","Type":"ContainerDied","Data":"5d08008d83bca220f0403dc80e7556b597d59221fbe3d7d98fde7ced0585f8e3"}
Sep 17 03:19:37.063808 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:37.063785    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:37.071241 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:37.071181    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk" podStartSLOduration=9.87592703 podStartE2EDuration="27.071166451s" totalImagesPullingTime="17.195239421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:37.069940198 +0000 UTC m=+27.673036203" watchObservedRunningTime="2026-09-17 03:19:37.071166451 +0000 UTC m=+27.674262456"
Sep 17 03:19:37.908757 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:37.908737    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:37.908864 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:37.908737    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:37.908922 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:37.908851    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:37.908982 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:37.908958    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:37.908982 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:37.908737    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:37.909089 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:37.909069    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:38.035402 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:38.035242    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mc2fv"]
Sep 17 03:19:38.038280 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:38.038217    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n5bhs"]
Sep 17 03:19:38.038937 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:38.038919    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5d8wb"]
Sep 17 03:19:38.055045 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:38.055020    2557 generic.go:356] "Generic (PLEG): container finished" podID="8257d8ba-f173-4c72-b229-b9cc0326cf73" containerID="93b72ef24de7d91844398e97f2d5ced130773ef36df4fc609b4559695d2325c8" exitCode=0
Sep 17 03:19:38.055484 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:38.055111    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6v5r" event={"ID":"8257d8ba-f173-4c72-b229-b9cc0326cf73","Type":"ContainerDied","Data":"93b72ef24de7d91844398e97f2d5ced130773ef36df4fc609b4559695d2325c8"}
Sep 17 03:19:38.055484 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:38.055162    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:38.055484 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:38.055250    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:38.055879 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:38.055747    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:38.055879 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:38.055833    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:38.055879 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:38.055847    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:38.055879 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:38.055871    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:38.056158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:38.055943    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:38.056158 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:38.056034    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:38.070536 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:38.070516    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:19:39.060068 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:39.060035    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6v5r" event={"ID":"8257d8ba-f173-4c72-b229-b9cc0326cf73","Type":"ContainerStarted","Data":"dc9351374d26ad0a93d0fe5eff75dc67d4a6e6fa071e8af85d26ab0c7d9331c8"}
Sep 17 03:19:39.908872 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:39.908800    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:39.909006 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:39.908876    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:39.909006 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:39.908978    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:39.909123 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:39.909093    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:39.909178 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:39.909126    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:39.909243 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:39.909220    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:40.064679 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:40.064649    2557 generic.go:356] "Generic (PLEG): container finished" podID="8257d8ba-f173-4c72-b229-b9cc0326cf73" containerID="dc9351374d26ad0a93d0fe5eff75dc67d4a6e6fa071e8af85d26ab0c7d9331c8" exitCode=0
Sep 17 03:19:40.065071 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:40.064722    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6v5r" event={"ID":"8257d8ba-f173-4c72-b229-b9cc0326cf73","Type":"ContainerDied","Data":"dc9351374d26ad0a93d0fe5eff75dc67d4a6e6fa071e8af85d26ab0c7d9331c8"}
Sep 17 03:19:41.909164 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:41.909133    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:41.909617 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:41.909134    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:41.909617 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:41.909142    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:41.909617 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:41.909276    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n5bhs" podUID="f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a"
Sep 17 03:19:41.909617 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:41.909403    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5d8wb" podUID="c7146f5b-e9f5-4469-8161-756c03f69dac"
Sep 17 03:19:41.909617 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:41.909483    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc2fv" podUID="c1b8633f-f1cc-440b-96e6-7e0b526f6583"
Sep 17 03:19:43.205966 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.205911    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-153.ec2.internal" event="NodeReady"
Sep 17 03:19:43.206301 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.206009    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 03:19:43.243863 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.243839    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c7659966b-292d6"]
Sep 17 03:19:43.292010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.291953    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4fqdd"]
Sep 17 03:19:43.292129 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.292074    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.294433 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.294411    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lh58r\""
Sep 17 03:19:43.294640 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.294455    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 03:19:43.294640 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.294564    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 03:19:43.294773 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.294751    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 03:19:43.300029 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.300009    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 03:19:43.305784 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.305768    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gm8dh"]
Sep 17 03:19:43.305890 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.305869    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:43.307951 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.307929    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 03:19:43.308056 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.307966    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8xsvg\""
Sep 17 03:19:43.308056 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.307974    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 03:19:43.327894 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.327823    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c7659966b-292d6"]
Sep 17 03:19:43.327894 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.327887    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gm8dh"
Sep 17 03:19:43.328021 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.327923    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4fqdd"]
Sep 17 03:19:43.328021 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.327949    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gm8dh"]
Sep 17 03:19:43.330046 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.330029    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 03:19:43.330179 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.330161    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 03:19:43.330266 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.330167    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 03:19:43.330266 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.330235    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tddkt\""
Sep 17 03:19:43.480135 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.480082    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:43.480135 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.480109    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f88b24fe-fd7e-4c77-8648-423434670be6-registry-certificates\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.480393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.480146    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-529pm\" (UniqueName: \"kubernetes.io/projected/7cbfac4e-1716-45d1-8a32-139bccea48a9-kube-api-access-529pm\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:43.480393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.480211    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.480393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.480256    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f88b24fe-fd7e-4c77-8648-423434670be6-trusted-ca\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.480393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.480304    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f88b24fe-fd7e-4c77-8648-423434670be6-ca-trust-extracted\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.480393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.480349    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cbfac4e-1716-45d1-8a32-139bccea48a9-config-volume\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:43.480635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.480403    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-bound-sa-token\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.480635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.480434    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f88b24fe-fd7e-4c77-8648-423434670be6-image-registry-private-configuration\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.480635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.480468    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/100b3459-13e3-4558-96ad-dd122ec25b08-cert\") pod \"ingress-canary-gm8dh\" (UID: \"100b3459-13e3-4558-96ad-dd122ec25b08\") " pod="openshift-ingress-canary/ingress-canary-gm8dh"
Sep 17 03:19:43.480635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.480492    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f88b24fe-fd7e-4c77-8648-423434670be6-installation-pull-secrets\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.480635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.480520    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7tgg\" (UniqueName: \"kubernetes.io/projected/100b3459-13e3-4558-96ad-dd122ec25b08-kube-api-access-t7tgg\") pod \"ingress-canary-gm8dh\" (UID: \"100b3459-13e3-4558-96ad-dd122ec25b08\") " pod="openshift-ingress-canary/ingress-canary-gm8dh"
Sep 17 03:19:43.480635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.480546    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cbfac4e-1716-45d1-8a32-139bccea48a9-tmp-dir\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:43.480635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.480569    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ggvd\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-kube-api-access-6ggvd\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.521677 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.521651    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gg8lm_af354256-5b4f-4a41-8391-b8145ea28913/konnectivity-agent/0.log" limit=1024
Sep 17 03:19:43.581511 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.581486    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cbfac4e-1716-45d1-8a32-139bccea48a9-tmp-dir\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:43.581609 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.581518    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ggvd\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-kube-api-access-6ggvd\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.581609 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.581551    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:43.581609 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.581579    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:43.581760 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:43.581656    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:43.581760 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:43.581668    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 03:19:43.581760 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:43.581713    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret podName:c1b8633f-f1cc-440b-96e6-7e0b526f6583 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:15.581697526 +0000 UTC m=+66.184793512 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret") pod "global-pull-secret-syncer-mc2fv" (UID: "c1b8633f-f1cc-440b-96e6-7e0b526f6583") : object "kube-system"/"original-pull-secret" not registered
Sep 17 03:19:43.581760 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:43.581734    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls podName:7cbfac4e-1716-45d1-8a32-139bccea48a9 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:44.08172343 +0000 UTC m=+34.684819422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls") pod "dns-default-4fqdd" (UID: "7cbfac4e-1716-45d1-8a32-139bccea48a9") : secret "dns-default-metrics-tls" not found
Sep 17 03:19:43.582010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.581766    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f88b24fe-fd7e-4c77-8648-423434670be6-registry-certificates\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.582010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.581813    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-529pm\" (UniqueName: \"kubernetes.io/projected/7cbfac4e-1716-45d1-8a32-139bccea48a9-kube-api-access-529pm\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:43.582010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.581837    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.582010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.581863    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f88b24fe-fd7e-4c77-8648-423434670be6-trusted-ca\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.582010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.581900    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f88b24fe-fd7e-4c77-8648-423434670be6-ca-trust-extracted\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.582010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.581920    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cbfac4e-1716-45d1-8a32-139bccea48a9-tmp-dir\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:43.582010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.581927    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs\") pod \"network-metrics-daemon-5d8wb\" (UID: \"c7146f5b-e9f5-4469-8161-756c03f69dac\") " pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:43.582010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.581951    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cbfac4e-1716-45d1-8a32-139bccea48a9-config-volume\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:43.582010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.581977    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-bound-sa-token\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.582010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.582008    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f88b24fe-fd7e-4c77-8648-423434670be6-image-registry-private-configuration\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.582551 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.582056    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/100b3459-13e3-4558-96ad-dd122ec25b08-cert\") pod \"ingress-canary-gm8dh\" (UID: \"100b3459-13e3-4558-96ad-dd122ec25b08\") " pod="openshift-ingress-canary/ingress-canary-gm8dh"
Sep 17 03:19:43.582551 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.582081    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f88b24fe-fd7e-4c77-8648-423434670be6-installation-pull-secrets\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.582551 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:43.582108    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:43.582551 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:43.582157    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs podName:c7146f5b-e9f5-4469-8161-756c03f69dac nodeName:}" failed. No retries permitted until 2026-09-17 03:20:15.582140446 +0000 UTC m=+66.185236427 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs") pod "network-metrics-daemon-5d8wb" (UID: "c7146f5b-e9f5-4469-8161-756c03f69dac") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 03:19:43.582551 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.582450    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f88b24fe-fd7e-4c77-8648-423434670be6-registry-certificates\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.582551 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.582531    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f88b24fe-fd7e-4c77-8648-423434670be6-ca-trust-extracted\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.582551 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:43.582544    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 03:19:43.582551 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:43.582557    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c7659966b-292d6: secret "image-registry-tls" not found
Sep 17 03:19:43.582927 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:43.582601    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls podName:f88b24fe-fd7e-4c77-8648-423434670be6 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:44.082584908 +0000 UTC m=+34.685680904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls") pod "image-registry-c7659966b-292d6" (UID: "f88b24fe-fd7e-4c77-8648-423434670be6") : secret "image-registry-tls" not found
Sep 17 03:19:43.582927 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.582107    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7tgg\" (UniqueName: \"kubernetes.io/projected/100b3459-13e3-4558-96ad-dd122ec25b08-kube-api-access-t7tgg\") pod \"ingress-canary-gm8dh\" (UID: \"100b3459-13e3-4558-96ad-dd122ec25b08\") " pod="openshift-ingress-canary/ingress-canary-gm8dh"
Sep 17 03:19:43.582927 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.582871    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f88b24fe-fd7e-4c77-8648-423434670be6-trusted-ca\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.582927 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.582899    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cbfac4e-1716-45d1-8a32-139bccea48a9-config-volume\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:43.587173 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.587148    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f88b24fe-fd7e-4c77-8648-423434670be6-image-registry-private-configuration\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.587173 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.587158    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/100b3459-13e3-4558-96ad-dd122ec25b08-cert\") pod \"ingress-canary-gm8dh\" (UID: \"100b3459-13e3-4558-96ad-dd122ec25b08\") " pod="openshift-ingress-canary/ingress-canary-gm8dh"
Sep 17 03:19:43.587409 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.587184    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f88b24fe-fd7e-4c77-8648-423434670be6-installation-pull-secrets\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.590596 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.590303    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-529pm\" (UniqueName: \"kubernetes.io/projected/7cbfac4e-1716-45d1-8a32-139bccea48a9-kube-api-access-529pm\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:43.590596 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.590556    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7tgg\" (UniqueName: \"kubernetes.io/projected/100b3459-13e3-4558-96ad-dd122ec25b08-kube-api-access-t7tgg\") pod \"ingress-canary-gm8dh\" (UID: \"100b3459-13e3-4558-96ad-dd122ec25b08\") " pod="openshift-ingress-canary/ingress-canary-gm8dh"
Sep 17 03:19:43.590976 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.590931    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ggvd\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-kube-api-access-6ggvd\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.591388 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.591370    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-bound-sa-token\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:43.606831 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.606792    2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gg8lm_af354256-5b4f-4a41-8391-b8145ea28913/konnectivity-agent/0.log" limit=1024
Sep 17 03:19:43.635744 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.635723    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gm8dh"
Sep 17 03:19:43.683542 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.683510    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c47g8\" (UniqueName: \"kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8\") pod \"network-check-target-n5bhs\" (UID: \"f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a\") " pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:43.683686 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:43.683667    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 03:19:43.683727 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:43.683691    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 03:19:43.683727 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:43.683701    2557 projected.go:196] Error preparing data for projected volume kube-api-access-c47g8 for pod openshift-network-diagnostics/network-check-target-n5bhs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:43.683799 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:43.683759    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8 podName:f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a nodeName:}" failed. No retries permitted until 2026-09-17 03:20:15.683745318 +0000 UTC m=+66.286841304 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-c47g8" (UniqueName: "kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8") pod "network-check-target-n5bhs" (UID: "f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 03:19:43.912905 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.912727    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:19:43.913070 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.912738    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:19:43.913070 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.912738    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:19:43.915609 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.915445    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 03:19:43.915609 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.915470    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 03:19:43.915609 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.915531    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d2k6b\""
Sep 17 03:19:43.915846 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.915810    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tmrnq\""
Sep 17 03:19:43.916117 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.915962    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 03:19:43.916117 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.916027    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 03:19:43.936180 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:43.936160    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gm8dh"]
Sep 17 03:19:44.086571 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:44.086536    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:44.086698 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:44.086595    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:44.086698 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:44.086648    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 03:19:44.086783 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:44.086718    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls podName:7cbfac4e-1716-45d1-8a32-139bccea48a9 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:45.086701159 +0000 UTC m=+35.689797148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls") pod "dns-default-4fqdd" (UID: "7cbfac4e-1716-45d1-8a32-139bccea48a9") : secret "dns-default-metrics-tls" not found
Sep 17 03:19:44.086783 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:44.086729    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 03:19:44.086783 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:44.086740    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c7659966b-292d6: secret "image-registry-tls" not found
Sep 17 03:19:44.086888 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:44.086788    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls podName:f88b24fe-fd7e-4c77-8648-423434670be6 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:45.086773458 +0000 UTC m=+35.689869455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls") pod "image-registry-c7659966b-292d6" (UID: "f88b24fe-fd7e-4c77-8648-423434670be6") : secret "image-registry-tls" not found
Sep 17 03:19:45.092419 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:45.092381    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:45.092888 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:45.092426    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:45.092888 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:45.092549    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 03:19:45.092888 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:45.092612    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls podName:7cbfac4e-1716-45d1-8a32-139bccea48a9 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:47.092596329 +0000 UTC m=+37.695692329 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls") pod "dns-default-4fqdd" (UID: "7cbfac4e-1716-45d1-8a32-139bccea48a9") : secret "dns-default-metrics-tls" not found
Sep 17 03:19:45.092888 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:45.092554    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 03:19:45.092888 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:45.092648    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c7659966b-292d6: secret "image-registry-tls" not found
Sep 17 03:19:45.092888 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:45.092702    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls podName:f88b24fe-fd7e-4c77-8648-423434670be6 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:47.092686589 +0000 UTC m=+37.695782587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls") pod "image-registry-c7659966b-292d6" (UID: "f88b24fe-fd7e-4c77-8648-423434670be6") : secret "image-registry-tls" not found
Sep 17 03:19:45.241858 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:19:45.241830    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100b3459_13e3_4558_96ad_dd122ec25b08.slice/crio-c1bbb874ecadf02d3bc0b8b62196a3a54dc1b3ad87816acc0f38da2e61b7f62a WatchSource:0}: Error finding container c1bbb874ecadf02d3bc0b8b62196a3a54dc1b3ad87816acc0f38da2e61b7f62a: Status 404 returned error can't find the container with id c1bbb874ecadf02d3bc0b8b62196a3a54dc1b3ad87816acc0f38da2e61b7f62a
Sep 17 03:19:46.080817 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:46.080748    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gm8dh" event={"ID":"100b3459-13e3-4558-96ad-dd122ec25b08","Type":"ContainerStarted","Data":"c1bbb874ecadf02d3bc0b8b62196a3a54dc1b3ad87816acc0f38da2e61b7f62a"}
Sep 17 03:19:46.084061 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:46.084023    2557 generic.go:356] "Generic (PLEG): container finished" podID="8257d8ba-f173-4c72-b229-b9cc0326cf73" containerID="fe2ab2c7f1aaa5a6fcd3a9d88bf68ab9ea10da6d28c9a8dd960c379c3a18f2aa" exitCode=0
Sep 17 03:19:46.084213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:46.084064    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6v5r" event={"ID":"8257d8ba-f173-4c72-b229-b9cc0326cf73","Type":"ContainerDied","Data":"fe2ab2c7f1aaa5a6fcd3a9d88bf68ab9ea10da6d28c9a8dd960c379c3a18f2aa"}
Sep 17 03:19:47.090290 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:47.090251    2557 generic.go:356] "Generic (PLEG): container finished" podID="8257d8ba-f173-4c72-b229-b9cc0326cf73" containerID="bd6a91f52ea46b6df0dad57b03f70d4ff0cd6f4521d55c8ed630b39a7ca29261" exitCode=0
Sep 17 03:19:47.091044 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:47.090336    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6v5r" event={"ID":"8257d8ba-f173-4c72-b229-b9cc0326cf73","Type":"ContainerDied","Data":"bd6a91f52ea46b6df0dad57b03f70d4ff0cd6f4521d55c8ed630b39a7ca29261"}
Sep 17 03:19:47.107102 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:47.107068    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:47.107241 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:47.107142    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:47.107424 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:47.107399    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 03:19:47.107541 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:47.107403    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 03:19:47.107541 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:47.107460    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls podName:7cbfac4e-1716-45d1-8a32-139bccea48a9 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:51.107445483 +0000 UTC m=+41.710541468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls") pod "dns-default-4fqdd" (UID: "7cbfac4e-1716-45d1-8a32-139bccea48a9") : secret "dns-default-metrics-tls" not found
Sep 17 03:19:47.107541 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:47.107470    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c7659966b-292d6: secret "image-registry-tls" not found
Sep 17 03:19:47.107541 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:47.107519    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls podName:f88b24fe-fd7e-4c77-8648-423434670be6 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:51.107502363 +0000 UTC m=+41.710598361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls") pod "image-registry-c7659966b-292d6" (UID: "f88b24fe-fd7e-4c77-8648-423434670be6") : secret "image-registry-tls" not found
Sep 17 03:19:48.095356 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:48.095217    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l6v5r" event={"ID":"8257d8ba-f173-4c72-b229-b9cc0326cf73","Type":"ContainerStarted","Data":"2cf3cbd55b5db829ab53e8698cdf19bae07f86defadfb5543f0fa17da3cd03f3"}
Sep 17 03:19:48.115886 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:48.115834    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l6v5r" podStartSLOduration=8.260491896 podStartE2EDuration="38.115792719s" totalImagesPullingTime="29.855300823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:48.114865488 +0000 UTC m=+38.717961503" watchObservedRunningTime="2026-09-17 03:19:48.115792719 +0000 UTC m=+38.718888724"
Sep 17 03:19:49.099799 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:49.099756    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gm8dh" event={"ID":"100b3459-13e3-4558-96ad-dd122ec25b08","Type":"ContainerStarted","Data":"9e6b7694a1c3fe41b9f86ca059676f58f3e56c557639cdd4f1f65a299f8f00a3"}
Sep 17 03:19:49.113784 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:49.113725    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gm8dh" podStartSLOduration=3.375338917 podStartE2EDuration="6.113704646s" totalImagesPullingTime="2.738365729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:19:49.112993922 +0000 UTC m=+39.716089926" watchObservedRunningTime="2026-09-17 03:19:49.113704646 +0000 UTC m=+39.716800651"
Sep 17 03:19:51.137682 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:51.137643    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:51.138128 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:51.137745    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:51.138128 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:51.137773    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 03:19:51.138128 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:51.137791    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c7659966b-292d6: secret "image-registry-tls" not found
Sep 17 03:19:51.138128 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:51.137844    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls podName:f88b24fe-fd7e-4c77-8648-423434670be6 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:59.137828744 +0000 UTC m=+49.740924730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls") pod "image-registry-c7659966b-292d6" (UID: "f88b24fe-fd7e-4c77-8648-423434670be6") : secret "image-registry-tls" not found
Sep 17 03:19:51.138128 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:51.137857    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 03:19:51.138128 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:51.137913    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls podName:7cbfac4e-1716-45d1-8a32-139bccea48a9 nodeName:}" failed. No retries permitted until 2026-09-17 03:19:59.137894832 +0000 UTC m=+49.740990815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls") pod "dns-default-4fqdd" (UID: "7cbfac4e-1716-45d1-8a32-139bccea48a9") : secret "dns-default-metrics-tls" not found
Sep 17 03:19:59.186027 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:59.185994    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:19:59.186547 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:19:59.186043    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:19:59.186547 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:59.186144    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 03:19:59.186547 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:59.186180    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 03:19:59.186547 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:59.186212    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c7659966b-292d6: secret "image-registry-tls" not found
Sep 17 03:19:59.186547 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:59.186224    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls podName:7cbfac4e-1716-45d1-8a32-139bccea48a9 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:15.186207869 +0000 UTC m=+65.789303855 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls") pod "dns-default-4fqdd" (UID: "7cbfac4e-1716-45d1-8a32-139bccea48a9") : secret "dns-default-metrics-tls" not found
Sep 17 03:19:59.186547 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:19:59.186265    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls podName:f88b24fe-fd7e-4c77-8648-423434670be6 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:15.186247545 +0000 UTC m=+65.789343573 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls") pod "image-registry-c7659966b-292d6" (UID: "f88b24fe-fd7e-4c77-8648-423434670be6") : secret "image-registry-tls" not found
Sep 17 03:20:04.856952 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:04.856918    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg"]
Sep 17 03:20:04.859303 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:04.859280    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg"
Sep 17 03:20:04.862290 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:04.862264    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 03:20:04.862437 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:04.862304    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 03:20:04.862437 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:04.862304    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 03:20:04.862437 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:04.862404    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 03:20:04.862997 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:04.862976    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-l2pfc\""
Sep 17 03:20:04.874332 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:04.874311    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg"]
Sep 17 03:20:04.923883 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:04.923861    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8afb8ac0-6b66-48d0-afc1-32750519d672-config\") pod \"service-ca-operator-cf7c776b7-kmwgg\" (UID: \"8afb8ac0-6b66-48d0-afc1-32750519d672\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg"
Sep 17 03:20:04.923993 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:04.923930    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8afb8ac0-6b66-48d0-afc1-32750519d672-serving-cert\") pod \"service-ca-operator-cf7c776b7-kmwgg\" (UID: \"8afb8ac0-6b66-48d0-afc1-32750519d672\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg"
Sep 17 03:20:04.924097 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:04.924012    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl7b6\" (UniqueName: \"kubernetes.io/projected/8afb8ac0-6b66-48d0-afc1-32750519d672-kube-api-access-kl7b6\") pod \"service-ca-operator-cf7c776b7-kmwgg\" (UID: \"8afb8ac0-6b66-48d0-afc1-32750519d672\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg"
Sep 17 03:20:05.024322 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.024290    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8afb8ac0-6b66-48d0-afc1-32750519d672-serving-cert\") pod \"service-ca-operator-cf7c776b7-kmwgg\" (UID: \"8afb8ac0-6b66-48d0-afc1-32750519d672\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg"
Sep 17 03:20:05.024411 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.024343    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl7b6\" (UniqueName: \"kubernetes.io/projected/8afb8ac0-6b66-48d0-afc1-32750519d672-kube-api-access-kl7b6\") pod \"service-ca-operator-cf7c776b7-kmwgg\" (UID: \"8afb8ac0-6b66-48d0-afc1-32750519d672\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg"
Sep 17 03:20:05.024411 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.024387    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8afb8ac0-6b66-48d0-afc1-32750519d672-config\") pod \"service-ca-operator-cf7c776b7-kmwgg\" (UID: \"8afb8ac0-6b66-48d0-afc1-32750519d672\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg"
Sep 17 03:20:05.024838 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.024819    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8afb8ac0-6b66-48d0-afc1-32750519d672-config\") pod \"service-ca-operator-cf7c776b7-kmwgg\" (UID: \"8afb8ac0-6b66-48d0-afc1-32750519d672\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg"
Sep 17 03:20:05.027878 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.027861    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8afb8ac0-6b66-48d0-afc1-32750519d672-serving-cert\") pod \"service-ca-operator-cf7c776b7-kmwgg\" (UID: \"8afb8ac0-6b66-48d0-afc1-32750519d672\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg"
Sep 17 03:20:05.032939 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.032916    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl7b6\" (UniqueName: \"kubernetes.io/projected/8afb8ac0-6b66-48d0-afc1-32750519d672-kube-api-access-kl7b6\") pod \"service-ca-operator-cf7c776b7-kmwgg\" (UID: \"8afb8ac0-6b66-48d0-afc1-32750519d672\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg"
Sep 17 03:20:05.056827 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.056803    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-qvxrt"]
Sep 17 03:20:05.059224 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.059209    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:05.061802 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.061786    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 03:20:05.062382 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.062365    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 03:20:05.062473 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.062442    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 03:20:05.065742 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.065720    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 03:20:05.065985 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.065955    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-776sm\""
Sep 17 03:20:05.066412 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.066376    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6685b49f78-sfnns"]
Sep 17 03:20:05.068590 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.068574    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.071430 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.071413    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 03:20:05.072426 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.072398    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 03:20:05.072517 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.072486    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jslhw\""
Sep 17 03:20:05.072961 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.072942    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 03:20:05.073054 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.073035    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 03:20:05.073362 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.073343    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 03:20:05.073454 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.073435    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 03:20:05.073538 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.073359    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 03:20:05.074152 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.074133    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-qvxrt"]
Sep 17 03:20:05.083225 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.083184    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6685b49f78-sfnns"]
Sep 17 03:20:05.125576 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.125556    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba4e66fe-1155-4f15-841f-b5eadceec705-trusted-ca\") pod \"console-operator-696884fb9c-qvxrt\" (UID: \"ba4e66fe-1155-4f15-841f-b5eadceec705\") " pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:05.125671 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.125593    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba4e66fe-1155-4f15-841f-b5eadceec705-config\") pod \"console-operator-696884fb9c-qvxrt\" (UID: \"ba4e66fe-1155-4f15-841f-b5eadceec705\") " pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:05.125671 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.125611    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj78c\" (UniqueName: \"kubernetes.io/projected/ba4e66fe-1155-4f15-841f-b5eadceec705-kube-api-access-nj78c\") pod \"console-operator-696884fb9c-qvxrt\" (UID: \"ba4e66fe-1155-4f15-841f-b5eadceec705\") " pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:05.125741 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.125706    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-stats-auth\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.125778 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.125741    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.125778 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.125760    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzpbq\" (UniqueName: \"kubernetes.io/projected/b1592c7c-2902-4b8d-9a15-b1bce854867f-kube-api-access-jzpbq\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.125853 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.125779    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba4e66fe-1155-4f15-841f-b5eadceec705-serving-cert\") pod \"console-operator-696884fb9c-qvxrt\" (UID: \"ba4e66fe-1155-4f15-841f-b5eadceec705\") " pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:05.125853 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.125795    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.125853 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.125827    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-default-certificate\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.169026 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.169006    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg"
Sep 17 03:20:05.226343 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.226313    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-stats-auth\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.226462 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.226377    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.226462 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.226406    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzpbq\" (UniqueName: \"kubernetes.io/projected/b1592c7c-2902-4b8d-9a15-b1bce854867f-kube-api-access-jzpbq\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.226462 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.226438    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba4e66fe-1155-4f15-841f-b5eadceec705-serving-cert\") pod \"console-operator-696884fb9c-qvxrt\" (UID: \"ba4e66fe-1155-4f15-841f-b5eadceec705\") " pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:05.226625 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.226463    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.226625 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.226490    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-default-certificate\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.226625 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.226582    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba4e66fe-1155-4f15-841f-b5eadceec705-trusted-ca\") pod \"console-operator-696884fb9c-qvxrt\" (UID: \"ba4e66fe-1155-4f15-841f-b5eadceec705\") " pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:05.226776 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.226623    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba4e66fe-1155-4f15-841f-b5eadceec705-config\") pod \"console-operator-696884fb9c-qvxrt\" (UID: \"ba4e66fe-1155-4f15-841f-b5eadceec705\") " pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:05.226776 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.226652    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj78c\" (UniqueName: \"kubernetes.io/projected/ba4e66fe-1155-4f15-841f-b5eadceec705-kube-api-access-nj78c\") pod \"console-operator-696884fb9c-qvxrt\" (UID: \"ba4e66fe-1155-4f15-841f-b5eadceec705\") " pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:05.226882 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:05.226860    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle podName:b1592c7c-2902-4b8d-9a15-b1bce854867f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:05.726836635 +0000 UTC m=+56.329932640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle") pod "router-default-6685b49f78-sfnns" (UID: "b1592c7c-2902-4b8d-9a15-b1bce854867f") : configmap references non-existent config key: service-ca.crt
Sep 17 03:20:05.226949 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:05.226935    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 03:20:05.227019 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:05.227001    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs podName:b1592c7c-2902-4b8d-9a15-b1bce854867f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:05.726984153 +0000 UTC m=+56.330080149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs") pod "router-default-6685b49f78-sfnns" (UID: "b1592c7c-2902-4b8d-9a15-b1bce854867f") : secret "router-metrics-certs-default" not found
Sep 17 03:20:05.227787 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.227760    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ba4e66fe-1155-4f15-841f-b5eadceec705-config\") pod \"console-operator-696884fb9c-qvxrt\" (UID: \"ba4e66fe-1155-4f15-841f-b5eadceec705\") " pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:05.227904 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.227878    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba4e66fe-1155-4f15-841f-b5eadceec705-trusted-ca\") pod \"console-operator-696884fb9c-qvxrt\" (UID: \"ba4e66fe-1155-4f15-841f-b5eadceec705\") " pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:05.229534 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.229483    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-default-certificate\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.229614 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.229541    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba4e66fe-1155-4f15-841f-b5eadceec705-serving-cert\") pod \"console-operator-696884fb9c-qvxrt\" (UID: \"ba4e66fe-1155-4f15-841f-b5eadceec705\") " pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:05.229909 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.229889    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-stats-auth\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.234784 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.234734    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzpbq\" (UniqueName: \"kubernetes.io/projected/b1592c7c-2902-4b8d-9a15-b1bce854867f-kube-api-access-jzpbq\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.234929 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.234907    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj78c\" (UniqueName: \"kubernetes.io/projected/ba4e66fe-1155-4f15-841f-b5eadceec705-kube-api-access-nj78c\") pod \"console-operator-696884fb9c-qvxrt\" (UID: \"ba4e66fe-1155-4f15-841f-b5eadceec705\") " pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:05.367395 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.367360    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:05.433748 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.433718    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg"]
Sep 17 03:20:05.436250 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:05.436222    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afb8ac0_6b66_48d0_afc1_32750519d672.slice/crio-7b4e0d98c63f01b65038f2a26d0e8d4b6d1d8d5072ed3ba5ef5740cc8ff663d5 WatchSource:0}: Error finding container 7b4e0d98c63f01b65038f2a26d0e8d4b6d1d8d5072ed3ba5ef5740cc8ff663d5: Status 404 returned error can't find the container with id 7b4e0d98c63f01b65038f2a26d0e8d4b6d1d8d5072ed3ba5ef5740cc8ff663d5
Sep 17 03:20:05.633317 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.633266    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-qvxrt"]
Sep 17 03:20:05.636128 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:05.636101    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4e66fe_1155_4f15_841f_b5eadceec705.slice/crio-66ef7730e8415fc5ba8f8b60f89a8df96a934617f572d9610b5cfd17b6ebabe6 WatchSource:0}: Error finding container 66ef7730e8415fc5ba8f8b60f89a8df96a934617f572d9610b5cfd17b6ebabe6: Status 404 returned error can't find the container with id 66ef7730e8415fc5ba8f8b60f89a8df96a934617f572d9610b5cfd17b6ebabe6
Sep 17 03:20:05.731143 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.731124    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.731261 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:05.731152    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:05.731309 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:05.731295    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle podName:b1592c7c-2902-4b8d-9a15-b1bce854867f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:06.731278644 +0000 UTC m=+57.334374630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle") pod "router-default-6685b49f78-sfnns" (UID: "b1592c7c-2902-4b8d-9a15-b1bce854867f") : configmap references non-existent config key: service-ca.crt
Sep 17 03:20:05.731309 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:05.731299    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 03:20:05.731378 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:05.731329    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs podName:b1592c7c-2902-4b8d-9a15-b1bce854867f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:06.731322609 +0000 UTC m=+57.334418643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs") pod "router-default-6685b49f78-sfnns" (UID: "b1592c7c-2902-4b8d-9a15-b1bce854867f") : secret "router-metrics-certs-default" not found
Sep 17 03:20:06.140124 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:06.140097    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt" event={"ID":"ba4e66fe-1155-4f15-841f-b5eadceec705","Type":"ContainerStarted","Data":"66ef7730e8415fc5ba8f8b60f89a8df96a934617f572d9610b5cfd17b6ebabe6"}
Sep 17 03:20:06.141376 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:06.141354    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg" event={"ID":"8afb8ac0-6b66-48d0-afc1-32750519d672","Type":"ContainerStarted","Data":"7b4e0d98c63f01b65038f2a26d0e8d4b6d1d8d5072ed3ba5ef5740cc8ff663d5"}
Sep 17 03:20:06.739635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:06.739601    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:06.739809 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:06.739648    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:06.739809 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:06.739780    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle podName:b1592c7c-2902-4b8d-9a15-b1bce854867f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:08.739758307 +0000 UTC m=+59.342854293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle") pod "router-default-6685b49f78-sfnns" (UID: "b1592c7c-2902-4b8d-9a15-b1bce854867f") : configmap references non-existent config key: service-ca.crt
Sep 17 03:20:06.739908 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:06.739838    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 03:20:06.739952 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:06.739915    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs podName:b1592c7c-2902-4b8d-9a15-b1bce854867f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:08.739896644 +0000 UTC m=+59.342992645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs") pod "router-default-6685b49f78-sfnns" (UID: "b1592c7c-2902-4b8d-9a15-b1bce854867f") : secret "router-metrics-certs-default" not found
Sep 17 03:20:08.753491 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:08.753446    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:08.753870 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:08.753511    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:08.753870 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:08.753619    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle podName:b1592c7c-2902-4b8d-9a15-b1bce854867f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:12.753599499 +0000 UTC m=+63.356695492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle") pod "router-default-6685b49f78-sfnns" (UID: "b1592c7c-2902-4b8d-9a15-b1bce854867f") : configmap references non-existent config key: service-ca.crt
Sep 17 03:20:08.753870 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:08.753676    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 03:20:08.753870 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:08.753734    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs podName:b1592c7c-2902-4b8d-9a15-b1bce854867f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:12.753719314 +0000 UTC m=+63.356815304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs") pod "router-default-6685b49f78-sfnns" (UID: "b1592c7c-2902-4b8d-9a15-b1bce854867f") : secret "router-metrics-certs-default" not found
Sep 17 03:20:09.150801 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:09.150773    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/0.log"
Sep 17 03:20:09.150913 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:09.150822    2557 generic.go:356] "Generic (PLEG): container finished" podID="ba4e66fe-1155-4f15-841f-b5eadceec705" containerID="9e608af21c0153a44637d6aa9a20e570ba855ee3bb98f7b3761eee44f0f8bd14" exitCode=255
Sep 17 03:20:09.150971 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:09.150912    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt" event={"ID":"ba4e66fe-1155-4f15-841f-b5eadceec705","Type":"ContainerDied","Data":"9e608af21c0153a44637d6aa9a20e570ba855ee3bb98f7b3761eee44f0f8bd14"}
Sep 17 03:20:09.151095 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:09.151079    2557 scope.go:117] "RemoveContainer" containerID="9e608af21c0153a44637d6aa9a20e570ba855ee3bb98f7b3761eee44f0f8bd14"
Sep 17 03:20:09.152528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:09.152505    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg" event={"ID":"8afb8ac0-6b66-48d0-afc1-32750519d672","Type":"ContainerStarted","Data":"3cfa0046a08875397000cfaee9478adc0c3917d628f1408c605c46b747fddaf1"}
Sep 17 03:20:09.177822 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:09.177785    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg" podStartSLOduration=1.610646891 podStartE2EDuration="5.177772851s" totalImagesPullingTime="3.56712596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:09.177203464 +0000 UTC m=+59.780299460" watchObservedRunningTime="2026-09-17 03:20:09.177772851 +0000 UTC m=+59.780868854"
Sep 17 03:20:10.074688 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:10.074659    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkpkk"
Sep 17 03:20:10.155572 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:10.155551    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/1.log"
Sep 17 03:20:10.155945 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:10.155930    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/0.log"
Sep 17 03:20:10.156023 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:10.155962    2557 generic.go:356] "Generic (PLEG): container finished" podID="ba4e66fe-1155-4f15-841f-b5eadceec705" containerID="2f92a7adfb8798c09c1ce38e41304ff2c7171ff93537a24d1fd6ced890dd9027" exitCode=255
Sep 17 03:20:10.156023 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:10.155989    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt" event={"ID":"ba4e66fe-1155-4f15-841f-b5eadceec705","Type":"ContainerDied","Data":"2f92a7adfb8798c09c1ce38e41304ff2c7171ff93537a24d1fd6ced890dd9027"}
Sep 17 03:20:10.156100 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:10.156051    2557 scope.go:117] "RemoveContainer" containerID="9e608af21c0153a44637d6aa9a20e570ba855ee3bb98f7b3761eee44f0f8bd14"
Sep 17 03:20:10.156232 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:10.156215    2557 scope.go:117] "RemoveContainer" containerID="2f92a7adfb8798c09c1ce38e41304ff2c7171ff93537a24d1fd6ced890dd9027"
Sep 17 03:20:10.156409 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:10.156385    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-qvxrt_openshift-console-operator(ba4e66fe-1155-4f15-841f-b5eadceec705)\"" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt" podUID="ba4e66fe-1155-4f15-841f-b5eadceec705"
Sep 17 03:20:11.160411 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:11.160381    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/1.log"
Sep 17 03:20:11.160794 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:11.160740    2557 scope.go:117] "RemoveContainer" containerID="2f92a7adfb8798c09c1ce38e41304ff2c7171ff93537a24d1fd6ced890dd9027"
Sep 17 03:20:11.160930 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:11.160907    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-qvxrt_openshift-console-operator(ba4e66fe-1155-4f15-841f-b5eadceec705)\"" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt" podUID="ba4e66fe-1155-4f15-841f-b5eadceec705"
Sep 17 03:20:11.905325 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:11.905293    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2cxq9"]
Sep 17 03:20:11.913750 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:11.913723    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:11.916507 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:11.916479    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 03:20:11.916655 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:11.916479    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 03:20:11.916655 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:11.916539    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 03:20:11.916655 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:11.916515    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 03:20:11.916926 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:11.916883    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-52cf6\""
Sep 17 03:20:11.917767 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:11.917740    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2cxq9"]
Sep 17 03:20:11.977044 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:11.977020    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18a22d55-ca40-44cb-87bc-2607b89f6385-data-volume\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:11.977139 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:11.977048    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18a22d55-ca40-44cb-87bc-2607b89f6385-crio-socket\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:11.977139 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:11.977100    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:11.977235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:11.977211    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:11.977271 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:11.977260    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkf5g\" (UniqueName: \"kubernetes.io/projected/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-api-access-xkf5g\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:12.077907 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:12.077884    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkf5g\" (UniqueName: \"kubernetes.io/projected/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-api-access-xkf5g\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:12.078006 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:12.077922    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18a22d55-ca40-44cb-87bc-2607b89f6385-data-volume\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:12.078006 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:12.077947    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18a22d55-ca40-44cb-87bc-2607b89f6385-crio-socket\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:12.078110 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:12.078037    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18a22d55-ca40-44cb-87bc-2607b89f6385-crio-socket\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:12.078173 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:12.078153    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:12.078230 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:12.078189    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18a22d55-ca40-44cb-87bc-2607b89f6385-data-volume\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:12.078270 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:12.078229    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 03:20:12.078270 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:12.078253    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:12.078347 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:12.078283    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm podName:18a22d55-ca40-44cb-87bc-2607b89f6385 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:12.57826504 +0000 UTC m=+63.181361036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2cxq9" (UID: "18a22d55-ca40-44cb-87bc-2607b89f6385") : configmap "kube-rbac-proxy" not found
Sep 17 03:20:12.078347 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:12.078325    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:12.078416 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:12.078380    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls podName:18a22d55-ca40-44cb-87bc-2607b89f6385 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:12.578367046 +0000 UTC m=+63.181463045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cxq9" (UID: "18a22d55-ca40-44cb-87bc-2607b89f6385") : secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:12.086124 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:12.086101    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkf5g\" (UniqueName: \"kubernetes.io/projected/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-api-access-xkf5g\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:12.580627 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:12.580591    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:12.581063 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:12.580660    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:12.581063 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:12.580732    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 03:20:12.581063 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:12.580767    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:12.581063 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:12.580824    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm podName:18a22d55-ca40-44cb-87bc-2607b89f6385 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:13.580803486 +0000 UTC m=+64.183899474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2cxq9" (UID: "18a22d55-ca40-44cb-87bc-2607b89f6385") : configmap "kube-rbac-proxy" not found
Sep 17 03:20:12.581063 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:12.580846    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls podName:18a22d55-ca40-44cb-87bc-2607b89f6385 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:13.580835957 +0000 UTC m=+64.183931944 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cxq9" (UID: "18a22d55-ca40-44cb-87bc-2607b89f6385") : secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:12.782367 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:12.782338    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:12.782479 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:12.782387    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:12.782540 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:12.782472    2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 03:20:12.782540 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:12.782527    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle podName:b1592c7c-2902-4b8d-9a15-b1bce854867f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:20.782514918 +0000 UTC m=+71.385610899 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle") pod "router-default-6685b49f78-sfnns" (UID: "b1592c7c-2902-4b8d-9a15-b1bce854867f") : configmap references non-existent config key: service-ca.crt
Sep 17 03:20:12.782621 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:12.782544    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs podName:b1592c7c-2902-4b8d-9a15-b1bce854867f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:20.782536207 +0000 UTC m=+71.385632188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs") pod "router-default-6685b49f78-sfnns" (UID: "b1592c7c-2902-4b8d-9a15-b1bce854867f") : secret "router-metrics-certs-default" not found
Sep 17 03:20:13.060379 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.060349    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rtsxl"]
Sep 17 03:20:13.063267 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.063250    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-rtsxl"
Sep 17 03:20:13.065275 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.065254    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 03:20:13.065442 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.065422    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 03:20:13.066112 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.066093    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-trn2f\""
Sep 17 03:20:13.066234 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.066096    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 03:20:13.066234 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.066097    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 03:20:13.071599 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.071576    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rtsxl"]
Sep 17 03:20:13.185928 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.185902    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvd9\" (UniqueName: \"kubernetes.io/projected/3e282781-b4dd-4c34-bd68-f427633c4e71-kube-api-access-jpvd9\") pod \"service-ca-568869fdc5-rtsxl\" (UID: \"3e282781-b4dd-4c34-bd68-f427633c4e71\") " pod="openshift-service-ca/service-ca-568869fdc5-rtsxl"
Sep 17 03:20:13.186041 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.185990    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3e282781-b4dd-4c34-bd68-f427633c4e71-signing-key\") pod \"service-ca-568869fdc5-rtsxl\" (UID: \"3e282781-b4dd-4c34-bd68-f427633c4e71\") " pod="openshift-service-ca/service-ca-568869fdc5-rtsxl"
Sep 17 03:20:13.186041 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.186013    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3e282781-b4dd-4c34-bd68-f427633c4e71-signing-cabundle\") pod \"service-ca-568869fdc5-rtsxl\" (UID: \"3e282781-b4dd-4c34-bd68-f427633c4e71\") " pod="openshift-service-ca/service-ca-568869fdc5-rtsxl"
Sep 17 03:20:13.286386 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.286358    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvd9\" (UniqueName: \"kubernetes.io/projected/3e282781-b4dd-4c34-bd68-f427633c4e71-kube-api-access-jpvd9\") pod \"service-ca-568869fdc5-rtsxl\" (UID: \"3e282781-b4dd-4c34-bd68-f427633c4e71\") " pod="openshift-service-ca/service-ca-568869fdc5-rtsxl"
Sep 17 03:20:13.286489 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.286432    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3e282781-b4dd-4c34-bd68-f427633c4e71-signing-key\") pod \"service-ca-568869fdc5-rtsxl\" (UID: \"3e282781-b4dd-4c34-bd68-f427633c4e71\") " pod="openshift-service-ca/service-ca-568869fdc5-rtsxl"
Sep 17 03:20:13.286489 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.286452    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3e282781-b4dd-4c34-bd68-f427633c4e71-signing-cabundle\") pod \"service-ca-568869fdc5-rtsxl\" (UID: \"3e282781-b4dd-4c34-bd68-f427633c4e71\") " pod="openshift-service-ca/service-ca-568869fdc5-rtsxl"
Sep 17 03:20:13.286992 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.286974    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3e282781-b4dd-4c34-bd68-f427633c4e71-signing-cabundle\") pod \"service-ca-568869fdc5-rtsxl\" (UID: \"3e282781-b4dd-4c34-bd68-f427633c4e71\") " pod="openshift-service-ca/service-ca-568869fdc5-rtsxl"
Sep 17 03:20:13.288990 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.288965    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3e282781-b4dd-4c34-bd68-f427633c4e71-signing-key\") pod \"service-ca-568869fdc5-rtsxl\" (UID: \"3e282781-b4dd-4c34-bd68-f427633c4e71\") " pod="openshift-service-ca/service-ca-568869fdc5-rtsxl"
Sep 17 03:20:13.298042 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.298017    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvd9\" (UniqueName: \"kubernetes.io/projected/3e282781-b4dd-4c34-bd68-f427633c4e71-kube-api-access-jpvd9\") pod \"service-ca-568869fdc5-rtsxl\" (UID: \"3e282781-b4dd-4c34-bd68-f427633c4e71\") " pod="openshift-service-ca/service-ca-568869fdc5-rtsxl"
Sep 17 03:20:13.372027 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.371998    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-rtsxl"
Sep 17 03:20:13.589360 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.589325    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:13.589673 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.589392    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:13.589673 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:13.589457    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 03:20:13.589673 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:13.589514    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:13.589673 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:13.589542    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm podName:18a22d55-ca40-44cb-87bc-2607b89f6385 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:15.589524371 +0000 UTC m=+66.192620368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2cxq9" (UID: "18a22d55-ca40-44cb-87bc-2607b89f6385") : configmap "kube-rbac-proxy" not found
Sep 17 03:20:13.589673 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:13.589557    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls podName:18a22d55-ca40-44cb-87bc-2607b89f6385 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:15.589551083 +0000 UTC m=+66.192647064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cxq9" (UID: "18a22d55-ca40-44cb-87bc-2607b89f6385") : secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:13.644778 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.644750    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rtsxl"]
Sep 17 03:20:13.647368 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:13.647341    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e282781_b4dd_4c34_bd68_f427633c4e71.slice/crio-79b54ab5a69f91f4ec87ef1dc6f5b9442728fa024b6d8c7109d878fb08d0bb48 WatchSource:0}: Error finding container 79b54ab5a69f91f4ec87ef1dc6f5b9442728fa024b6d8c7109d878fb08d0bb48: Status 404 returned error can't find the container with id 79b54ab5a69f91f4ec87ef1dc6f5b9442728fa024b6d8c7109d878fb08d0bb48
Sep 17 03:20:13.665855 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:13.665835    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b77f8_cf71bc53-8158-46f9-9ad2-9380beca6a48/dns-node-resolver/0.log"
Sep 17 03:20:14.168502 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:14.168473    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-rtsxl" event={"ID":"3e282781-b4dd-4c34-bd68-f427633c4e71","Type":"ContainerStarted","Data":"ca8d67829ec3a3933f04ba63910fc155f7ded8bb2cdf4cb6292d95bf1ae3ea49"}
Sep 17 03:20:14.168645 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:14.168516    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-rtsxl" event={"ID":"3e282781-b4dd-4c34-bd68-f427633c4e71","Type":"ContainerStarted","Data":"79b54ab5a69f91f4ec87ef1dc6f5b9442728fa024b6d8c7109d878fb08d0bb48"}
Sep 17 03:20:14.186879 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:14.186831    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-rtsxl" podStartSLOduration=1.186817369 podStartE2EDuration="1.186817369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:14.185944694 +0000 UTC m=+64.789040698" watchObservedRunningTime="2026-09-17 03:20:14.186817369 +0000 UTC m=+64.789913372"
Sep 17 03:20:14.665965 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:14.665943    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f6gqp_c9654da7-e49f-41b1-b764-41002e5bdb28/node-ca/0.log"
Sep 17 03:20:15.203249 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.203216    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:20:15.203442 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.203296    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:20:15.203923 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:15.203651    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 03:20:15.203923 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:15.203719    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls podName:7cbfac4e-1716-45d1-8a32-139bccea48a9 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:47.20370039 +0000 UTC m=+97.806796388 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls") pod "dns-default-4fqdd" (UID: "7cbfac4e-1716-45d1-8a32-139bccea48a9") : secret "dns-default-metrics-tls" not found
Sep 17 03:20:15.204212 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:15.204183    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 03:20:15.204310 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:15.204300    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c7659966b-292d6: secret "image-registry-tls" not found
Sep 17 03:20:15.204477 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:15.204455    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls podName:f88b24fe-fd7e-4c77-8648-423434670be6 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:47.204439874 +0000 UTC m=+97.807535870 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls") pod "image-registry-c7659966b-292d6" (UID: "f88b24fe-fd7e-4c77-8648-423434670be6") : secret "image-registry-tls" not found
Sep 17 03:20:15.367636 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.367612    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:15.367758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.367669    2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:15.368029 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.368017    2557 scope.go:117] "RemoveContainer" containerID="2f92a7adfb8798c09c1ce38e41304ff2c7171ff93537a24d1fd6ced890dd9027"
Sep 17 03:20:15.368204 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:15.368169    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-qvxrt_openshift-console-operator(ba4e66fe-1155-4f15-841f-b5eadceec705)\"" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt" podUID="ba4e66fe-1155-4f15-841f-b5eadceec705"
Sep 17 03:20:15.609793 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.609727    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:15.609793 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.609766    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:20:15.609987 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.609794    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs\") pod \"network-metrics-daemon-5d8wb\" (UID: \"c7146f5b-e9f5-4469-8161-756c03f69dac\") " pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:20:15.609987 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.609853    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:15.609987 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:15.609900    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:15.609987 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:15.609955    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 03:20:15.609987 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:15.609962    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls podName:18a22d55-ca40-44cb-87bc-2607b89f6385 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:19.609944322 +0000 UTC m=+70.213040307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cxq9" (UID: "18a22d55-ca40-44cb-87bc-2607b89f6385") : secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:15.610273 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:15.610001    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm podName:18a22d55-ca40-44cb-87bc-2607b89f6385 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:19.609986215 +0000 UTC m=+70.213082198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2cxq9" (UID: "18a22d55-ca40-44cb-87bc-2607b89f6385") : configmap "kube-rbac-proxy" not found
Sep 17 03:20:15.611988 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.611968    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 03:20:15.612154 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.612137    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 03:20:15.620400 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:15.620381    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 03:20:15.620446 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:15.620425    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs podName:c7146f5b-e9f5-4469-8161-756c03f69dac nodeName:}" failed. No retries permitted until 2026-09-17 03:21:19.620413838 +0000 UTC m=+130.223509825 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs") pod "network-metrics-daemon-5d8wb" (UID: "c7146f5b-e9f5-4469-8161-756c03f69dac") : secret "metrics-daemon-secret" not found
Sep 17 03:20:15.622413 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.622390    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c1b8633f-f1cc-440b-96e6-7e0b526f6583-original-pull-secret\") pod \"global-pull-secret-syncer-mc2fv\" (UID: \"c1b8633f-f1cc-440b-96e6-7e0b526f6583\") " pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:20:15.667109 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.667090    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gm8dh_100b3459-13e3-4558-96ad-dd122ec25b08/serve-healthcheck-canary/0.log"
Sep 17 03:20:15.710553 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.710531    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c47g8\" (UniqueName: \"kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8\") pod \"network-check-target-n5bhs\" (UID: \"f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a\") " pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:20:15.712470 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.712456    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 03:20:15.722771 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.722754    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 03:20:15.731738 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.731721    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc2fv"
Sep 17 03:20:15.733354 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:15.733333    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c47g8\" (UniqueName: \"kubernetes.io/projected/f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a-kube-api-access-c47g8\") pod \"network-check-target-n5bhs\" (UID: \"f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a\") " pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:20:16.025154 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:16.025129    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mc2fv"]
Sep 17 03:20:16.027086 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:16.027066    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d2k6b\""
Sep 17 03:20:16.027793 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:16.027769    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b8633f_f1cc_440b_96e6_7e0b526f6583.slice/crio-040ae75a23ea0585804919d486e31e6375dbd8eac4ca14e4b000e2b2486d7e98 WatchSource:0}: Error finding container 040ae75a23ea0585804919d486e31e6375dbd8eac4ca14e4b000e2b2486d7e98: Status 404 returned error can't find the container with id 040ae75a23ea0585804919d486e31e6375dbd8eac4ca14e4b000e2b2486d7e98
Sep 17 03:20:16.035140 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:16.035122    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:20:16.175140 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:16.175109    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mc2fv" event={"ID":"c1b8633f-f1cc-440b-96e6-7e0b526f6583","Type":"ContainerStarted","Data":"040ae75a23ea0585804919d486e31e6375dbd8eac4ca14e4b000e2b2486d7e98"}
Sep 17 03:20:16.305746 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:16.305687    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n5bhs"]
Sep 17 03:20:16.308472 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:16.308445    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bbbf70_fe1a_4a2b_ad64_b7d88b9d798a.slice/crio-a40097ea31e05376c11d9039d72abe0eddf902e15fbffc1e12839e5638f37cf3 WatchSource:0}: Error finding container a40097ea31e05376c11d9039d72abe0eddf902e15fbffc1e12839e5638f37cf3: Status 404 returned error can't find the container with id a40097ea31e05376c11d9039d72abe0eddf902e15fbffc1e12839e5638f37cf3
Sep 17 03:20:17.180486 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:17.180452    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n5bhs" event={"ID":"f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a","Type":"ContainerStarted","Data":"a40097ea31e05376c11d9039d72abe0eddf902e15fbffc1e12839e5638f37cf3"}
Sep 17 03:20:19.643872 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:19.643839    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:19.644507 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:19.643912    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:19.644507 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:19.644021    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:19.644507 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:19.644081    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls podName:18a22d55-ca40-44cb-87bc-2607b89f6385 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:27.644063594 +0000 UTC m=+78.247159577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2cxq9" (UID: "18a22d55-ca40-44cb-87bc-2607b89f6385") : secret "insights-runtime-extractor-tls" not found
Sep 17 03:20:19.644507 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:19.644285    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 03:20:19.644507 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:19.644358    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm podName:18a22d55-ca40-44cb-87bc-2607b89f6385 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:27.644339724 +0000 UTC m=+78.247435720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2cxq9" (UID: "18a22d55-ca40-44cb-87bc-2607b89f6385") : configmap "kube-rbac-proxy" not found
Sep 17 03:20:20.853396 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:20.853359    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:20.853770 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:20.853414    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:20.853770 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:20.853527    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle podName:b1592c7c-2902-4b8d-9a15-b1bce854867f nodeName:}" failed. No retries permitted until 2026-09-17 03:20:36.853513939 +0000 UTC m=+87.456609920 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle") pod "router-default-6685b49f78-sfnns" (UID: "b1592c7c-2902-4b8d-9a15-b1bce854867f") : configmap references non-existent config key: service-ca.crt
Sep 17 03:20:20.855662 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:20.855611    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1592c7c-2902-4b8d-9a15-b1bce854867f-metrics-certs\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:21.192924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:21.192895    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n5bhs" event={"ID":"f9bbbf70-fe1a-4a2b-ad64-b7d88b9d798a","Type":"ContainerStarted","Data":"12f81a096aab2a0e9fec2285b58534e284df897b750341745ded30d7e0f7f161"}
Sep 17 03:20:21.193097 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:21.193084    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:20:21.194353 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:21.194328    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mc2fv" event={"ID":"c1b8633f-f1cc-440b-96e6-7e0b526f6583","Type":"ContainerStarted","Data":"ab45713e449092f29dc8312ca9f038953e2ffae1561bcc3c454e0f4dbc1c40df"}
Sep 17 03:20:21.206699 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:21.206662    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n5bhs" podStartSLOduration=67.606780278 podStartE2EDuration="1m12.206648243s" totalImagesPullingTime="4.599867965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:21.205297632 +0000 UTC m=+71.808393638" watchObservedRunningTime="2026-09-17 03:20:21.206648243 +0000 UTC m=+71.809744238"
Sep 17 03:20:27.705698 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:27.705648    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:27.706224 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:27.705741    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:27.706307 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:27.706285    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18a22d55-ca40-44cb-87bc-2607b89f6385-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:27.707995 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:27.707966    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18a22d55-ca40-44cb-87bc-2607b89f6385-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2cxq9\" (UID: \"18a22d55-ca40-44cb-87bc-2607b89f6385\") " pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:27.823928 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:27.823895    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2cxq9"
Sep 17 03:20:28.091986 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:28.091814    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mc2fv" podStartSLOduration=73.207567612 podStartE2EDuration="1m18.09179314s" totalImagesPullingTime="4.884225528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:21.216732374 +0000 UTC m=+71.819828377" watchObservedRunningTime="2026-09-17 03:20:28.09179314 +0000 UTC m=+78.694889147"
Sep 17 03:20:28.092601 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:28.092578    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2cxq9"]
Sep 17 03:20:28.095687 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:28.095660    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a22d55_ca40_44cb_87bc_2607b89f6385.slice/crio-7050023cf02f86d19d9bf78a058085eda44e2032150900a95e78d2e39dab3625 WatchSource:0}: Error finding container 7050023cf02f86d19d9bf78a058085eda44e2032150900a95e78d2e39dab3625: Status 404 returned error can't find the container with id 7050023cf02f86d19d9bf78a058085eda44e2032150900a95e78d2e39dab3625
Sep 17 03:20:28.218244 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:28.218220    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2cxq9" event={"ID":"18a22d55-ca40-44cb-87bc-2607b89f6385","Type":"ContainerStarted","Data":"0d24cb71dfadf53fc63564de9fcaf6196578d0940bc5a3860b4c9a966b649c28"}
Sep 17 03:20:28.218338 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:28.218275    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2cxq9" event={"ID":"18a22d55-ca40-44cb-87bc-2607b89f6385","Type":"ContainerStarted","Data":"7050023cf02f86d19d9bf78a058085eda44e2032150900a95e78d2e39dab3625"}
Sep 17 03:20:29.223924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:29.223856    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2cxq9" event={"ID":"18a22d55-ca40-44cb-87bc-2607b89f6385","Type":"ContainerStarted","Data":"f18106314d245b6779925dd9f2f45ce5e2cd94de087ca13f25a3fe3cf6e0f328"}
Sep 17 03:20:29.912690 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:29.912665    2557 scope.go:117] "RemoveContainer" containerID="2f92a7adfb8798c09c1ce38e41304ff2c7171ff93537a24d1fd6ced890dd9027"
Sep 17 03:20:30.039762 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:30.039723    2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4e66fe_1155_4f15_841f_b5eadceec705.slice/crio-conmon-95cb6c8d2cf474ee4db604f2f09cf338c1341f2d81be96b317dee8d80c08f4dd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4e66fe_1155_4f15_841f_b5eadceec705.slice/crio-95cb6c8d2cf474ee4db604f2f09cf338c1341f2d81be96b317dee8d80c08f4dd.scope\": RecentStats: unable to find data in memory cache]"
Sep 17 03:20:30.229294 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:30.229229    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/1.log"
Sep 17 03:20:30.229294 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:30.229277    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt" event={"ID":"ba4e66fe-1155-4f15-841f-b5eadceec705","Type":"ContainerStarted","Data":"95cb6c8d2cf474ee4db604f2f09cf338c1341f2d81be96b317dee8d80c08f4dd"}
Sep 17 03:20:30.229739 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:30.229643    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:30.231138 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:30.231113    2557 patch_prober.go:28] interesting pod/console-operator-696884fb9c-qvxrt container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused" start-of-body=
Sep 17 03:20:30.231281 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:30.231161    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt" podUID="ba4e66fe-1155-4f15-841f-b5eadceec705" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.10:8443/readyz\": dial tcp 10.132.0.10:8443: connect: connection refused"
Sep 17 03:20:30.247072 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:30.247023    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt" podStartSLOduration=21.87764099 podStartE2EDuration="25.24700609s" totalImagesPullingTime="3.3693651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:30.246076444 +0000 UTC m=+80.849172450" watchObservedRunningTime="2026-09-17 03:20:30.24700609 +0000 UTC m=+80.850102096"
Sep 17 03:20:31.235077 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:31.235039    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2cxq9" event={"ID":"18a22d55-ca40-44cb-87bc-2607b89f6385","Type":"ContainerStarted","Data":"393d94d6af9f97b782b73a7ccf9483e81ab0affeb49ee1da926a5d2f4242c45a"}
Sep 17 03:20:31.236980 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:31.236959    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:20:31.237483 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:31.237461    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/1.log"
Sep 17 03:20:31.237601 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:31.237500    2557 generic.go:356] "Generic (PLEG): container finished" podID="ba4e66fe-1155-4f15-841f-b5eadceec705" containerID="95cb6c8d2cf474ee4db604f2f09cf338c1341f2d81be96b317dee8d80c08f4dd" exitCode=255
Sep 17 03:20:31.237601 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:31.237531    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt" event={"ID":"ba4e66fe-1155-4f15-841f-b5eadceec705","Type":"ContainerDied","Data":"95cb6c8d2cf474ee4db604f2f09cf338c1341f2d81be96b317dee8d80c08f4dd"}
Sep 17 03:20:31.237725 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:31.237607    2557 scope.go:117] "RemoveContainer" containerID="2f92a7adfb8798c09c1ce38e41304ff2c7171ff93537a24d1fd6ced890dd9027"
Sep 17 03:20:31.237777 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:31.237747    2557 scope.go:117] "RemoveContainer" containerID="95cb6c8d2cf474ee4db604f2f09cf338c1341f2d81be96b317dee8d80c08f4dd"
Sep 17 03:20:31.237972 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:31.237943    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-qvxrt_openshift-console-operator(ba4e66fe-1155-4f15-841f-b5eadceec705)\"" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt" podUID="ba4e66fe-1155-4f15-841f-b5eadceec705"
Sep 17 03:20:31.254422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:31.254350    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2cxq9" podStartSLOduration=18.058312941 podStartE2EDuration="20.254334044s" totalImagesPullingTime="2.196021103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 03:20:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:31.25354412 +0000 UTC m=+81.856640125" watchObservedRunningTime="2026-09-17 03:20:31.254334044 +0000 UTC m=+81.857430050"
Sep 17 03:20:32.242048 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:32.242019    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:20:32.242433 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:32.242396    2557 scope.go:117] "RemoveContainer" containerID="95cb6c8d2cf474ee4db604f2f09cf338c1341f2d81be96b317dee8d80c08f4dd"
Sep 17 03:20:32.242614 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:32.242589    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-qvxrt_openshift-console-operator(ba4e66fe-1155-4f15-841f-b5eadceec705)\"" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt" podUID="ba4e66fe-1155-4f15-841f-b5eadceec705"
Sep 17 03:20:35.367611 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:35.367578    2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:20:35.368016 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:35.367925    2557 scope.go:117] "RemoveContainer" containerID="95cb6c8d2cf474ee4db604f2f09cf338c1341f2d81be96b317dee8d80c08f4dd"
Sep 17 03:20:35.368096 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:35.368074    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-qvxrt_openshift-console-operator(ba4e66fe-1155-4f15-841f-b5eadceec705)\"" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt" podUID="ba4e66fe-1155-4f15-841f-b5eadceec705"
Sep 17 03:20:36.869454 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:36.869419    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:36.870045 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:36.870017    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1592c7c-2902-4b8d-9a15-b1bce854867f-service-ca-bundle\") pod \"router-default-6685b49f78-sfnns\" (UID: \"b1592c7c-2902-4b8d-9a15-b1bce854867f\") " pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:36.879467 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:36.879446    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jslhw\""
Sep 17 03:20:36.887925 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:36.887906    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:37.161652 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:37.161580    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6685b49f78-sfnns"]
Sep 17 03:20:37.165475 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:37.165450    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1592c7c_2902_4b8d_9a15_b1bce854867f.slice/crio-43b5c39dc59cf851e47dde14a98474a5cf9bfa71432b5694dbcc06dae391b007 WatchSource:0}: Error finding container 43b5c39dc59cf851e47dde14a98474a5cf9bfa71432b5694dbcc06dae391b007: Status 404 returned error can't find the container with id 43b5c39dc59cf851e47dde14a98474a5cf9bfa71432b5694dbcc06dae391b007
Sep 17 03:20:37.259338 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:37.259311    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6685b49f78-sfnns" event={"ID":"b1592c7c-2902-4b8d-9a15-b1bce854867f","Type":"ContainerStarted","Data":"75d15693fcaf6c5e348a0e74032d2db381bbc8ba01e25cb53854bb3c9728fc19"}
Sep 17 03:20:37.259504 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:37.259355    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6685b49f78-sfnns" event={"ID":"b1592c7c-2902-4b8d-9a15-b1bce854867f","Type":"ContainerStarted","Data":"43b5c39dc59cf851e47dde14a98474a5cf9bfa71432b5694dbcc06dae391b007"}
Sep 17 03:20:37.275879 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:37.275834    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6685b49f78-sfnns" podStartSLOduration=32.275819559 podStartE2EDuration="32.275819559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:37.274707017 +0000 UTC m=+87.877803020" watchObservedRunningTime="2026-09-17 03:20:37.275819559 +0000 UTC m=+87.878915562"
Sep 17 03:20:37.888438 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:37.888400    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:37.892445 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:37.892416    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:38.262984 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:38.262917    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:38.264235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:38.264208    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6685b49f78-sfnns"
Sep 17 03:20:39.789394 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:39.789359    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7kkzg"]
Sep 17 03:20:39.792609 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:39.792589    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7kkzg"
Sep 17 03:20:39.794896 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:39.794877    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lnfzp\""
Sep 17 03:20:39.795150 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:39.795123    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 03:20:39.800766 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:39.800743    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7kkzg"]
Sep 17 03:20:39.891706 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:39.891684    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a6bfc99c-f10f-452d-a620-c2cb9e34125e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7kkzg\" (UID: \"a6bfc99c-f10f-452d-a620-c2cb9e34125e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7kkzg"
Sep 17 03:20:39.992079 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:39.992056    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a6bfc99c-f10f-452d-a620-c2cb9e34125e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7kkzg\" (UID: \"a6bfc99c-f10f-452d-a620-c2cb9e34125e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7kkzg"
Sep 17 03:20:39.992185 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:39.992166    2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 03:20:39.992258 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:39.992245    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6bfc99c-f10f-452d-a620-c2cb9e34125e-tls-certificates podName:a6bfc99c-f10f-452d-a620-c2cb9e34125e nodeName:}" failed. No retries permitted until 2026-09-17 03:20:40.492229323 +0000 UTC m=+91.095325309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a6bfc99c-f10f-452d-a620-c2cb9e34125e-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-7kkzg" (UID: "a6bfc99c-f10f-452d-a620-c2cb9e34125e") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 03:20:40.495481 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:40.495452    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a6bfc99c-f10f-452d-a620-c2cb9e34125e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7kkzg\" (UID: \"a6bfc99c-f10f-452d-a620-c2cb9e34125e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7kkzg"
Sep 17 03:20:40.497864 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:40.497837    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a6bfc99c-f10f-452d-a620-c2cb9e34125e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7kkzg\" (UID: \"a6bfc99c-f10f-452d-a620-c2cb9e34125e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7kkzg"
Sep 17 03:20:40.702147 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:40.702121    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7kkzg"
Sep 17 03:20:40.976792 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:40.976763    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7kkzg"]
Sep 17 03:20:40.979551 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:40.979525    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6bfc99c_f10f_452d_a620_c2cb9e34125e.slice/crio-ffdfeade39e776371f3054110e8c8b71f55544c42a757f932fe3126724ad243c WatchSource:0}: Error finding container ffdfeade39e776371f3054110e8c8b71f55544c42a757f932fe3126724ad243c: Status 404 returned error can't find the container with id ffdfeade39e776371f3054110e8c8b71f55544c42a757f932fe3126724ad243c
Sep 17 03:20:41.279262 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:41.279227    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7kkzg" event={"ID":"a6bfc99c-f10f-452d-a620-c2cb9e34125e","Type":"ContainerStarted","Data":"ffdfeade39e776371f3054110e8c8b71f55544c42a757f932fe3126724ad243c"}
Sep 17 03:20:42.283858 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:42.283829    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7kkzg" event={"ID":"a6bfc99c-f10f-452d-a620-c2cb9e34125e","Type":"ContainerStarted","Data":"125a4ea7ab233f0494de558e72884df05ddfb31e42902e05c162df02f6130c2a"}
Sep 17 03:20:42.284139 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:42.283959    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7kkzg"
Sep 17 03:20:42.288419 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:42.288399    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7kkzg"
Sep 17 03:20:42.300550 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:42.300512    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7kkzg" podStartSLOduration=2.138847129 podStartE2EDuration="3.300498279s" totalImagesPullingTime="1.16165115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:42.300171772 +0000 UTC m=+92.903267778" watchObservedRunningTime="2026-09-17 03:20:42.300498279 +0000 UTC m=+92.903594282"
Sep 17 03:20:42.849059 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:42.849025    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"]
Sep 17 03:20:42.882981 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:42.882951    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"]
Sep 17 03:20:42.883097 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:42.883047    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"
Sep 17 03:20:42.885606 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:42.885579    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 03:20:42.885719 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:42.885625    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 03:20:42.885719 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:42.885697    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 03:20:42.886214 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:42.886175    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zwwl9\""
Sep 17 03:20:42.886214 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:42.886181    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 03:20:42.886397 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:42.886241    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 03:20:43.012779 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:43.012755    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/faba1bc0-5c90-49c7-9fbf-b2a81a4d3516-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-bqnhq\" (UID: \"faba1bc0-5c90-49c7-9fbf-b2a81a4d3516\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"
Sep 17 03:20:43.012902 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:43.012784    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faba1bc0-5c90-49c7-9fbf-b2a81a4d3516-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-bqnhq\" (UID: \"faba1bc0-5c90-49c7-9fbf-b2a81a4d3516\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"
Sep 17 03:20:43.012902 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:43.012805    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l66qk\" (UniqueName: \"kubernetes.io/projected/faba1bc0-5c90-49c7-9fbf-b2a81a4d3516-kube-api-access-l66qk\") pod \"prometheus-operator-68bcd7f987-bqnhq\" (UID: \"faba1bc0-5c90-49c7-9fbf-b2a81a4d3516\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"
Sep 17 03:20:43.012902 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:43.012834    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faba1bc0-5c90-49c7-9fbf-b2a81a4d3516-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-bqnhq\" (UID: \"faba1bc0-5c90-49c7-9fbf-b2a81a4d3516\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"
Sep 17 03:20:43.113411 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:43.113338    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faba1bc0-5c90-49c7-9fbf-b2a81a4d3516-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-bqnhq\" (UID: \"faba1bc0-5c90-49c7-9fbf-b2a81a4d3516\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"
Sep 17 03:20:43.113519 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:43.113440    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/faba1bc0-5c90-49c7-9fbf-b2a81a4d3516-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-bqnhq\" (UID: \"faba1bc0-5c90-49c7-9fbf-b2a81a4d3516\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"
Sep 17 03:20:43.113519 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:43.113476    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faba1bc0-5c90-49c7-9fbf-b2a81a4d3516-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-bqnhq\" (UID: \"faba1bc0-5c90-49c7-9fbf-b2a81a4d3516\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"
Sep 17 03:20:43.113519 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:43.113504    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l66qk\" (UniqueName: \"kubernetes.io/projected/faba1bc0-5c90-49c7-9fbf-b2a81a4d3516-kube-api-access-l66qk\") pod \"prometheus-operator-68bcd7f987-bqnhq\" (UID: \"faba1bc0-5c90-49c7-9fbf-b2a81a4d3516\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"
Sep 17 03:20:43.113927 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:43.113907    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/faba1bc0-5c90-49c7-9fbf-b2a81a4d3516-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-bqnhq\" (UID: \"faba1bc0-5c90-49c7-9fbf-b2a81a4d3516\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"
Sep 17 03:20:43.115929 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:43.115912    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/faba1bc0-5c90-49c7-9fbf-b2a81a4d3516-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-bqnhq\" (UID: \"faba1bc0-5c90-49c7-9fbf-b2a81a4d3516\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"
Sep 17 03:20:43.115929 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:43.115925    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/faba1bc0-5c90-49c7-9fbf-b2a81a4d3516-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-bqnhq\" (UID: \"faba1bc0-5c90-49c7-9fbf-b2a81a4d3516\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"
Sep 17 03:20:43.122813 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:43.122788    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l66qk\" (UniqueName: \"kubernetes.io/projected/faba1bc0-5c90-49c7-9fbf-b2a81a4d3516-kube-api-access-l66qk\") pod \"prometheus-operator-68bcd7f987-bqnhq\" (UID: \"faba1bc0-5c90-49c7-9fbf-b2a81a4d3516\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"
Sep 17 03:20:43.191713 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:43.191693    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"
Sep 17 03:20:43.469123 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:43.469028    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq"]
Sep 17 03:20:43.471438 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:43.471418    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaba1bc0_5c90_49c7_9fbf_b2a81a4d3516.slice/crio-bd1bc84eaa3915e5e9092047b04891df55c937b5c11007f1843005bda8f5104b WatchSource:0}: Error finding container bd1bc84eaa3915e5e9092047b04891df55c937b5c11007f1843005bda8f5104b: Status 404 returned error can't find the container with id bd1bc84eaa3915e5e9092047b04891df55c937b5c11007f1843005bda8f5104b
Sep 17 03:20:44.290996 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:44.290966    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq" event={"ID":"faba1bc0-5c90-49c7-9fbf-b2a81a4d3516","Type":"ContainerStarted","Data":"bd1bc84eaa3915e5e9092047b04891df55c937b5c11007f1843005bda8f5104b"}
Sep 17 03:20:45.296214 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:45.296107    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq" event={"ID":"faba1bc0-5c90-49c7-9fbf-b2a81a4d3516","Type":"ContainerStarted","Data":"f931ec8ace39cd08b8b86b9174421a697a556ab20983a5e2b35d4cd761dd1ac6"}
Sep 17 03:20:45.296214 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:45.296172    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq" event={"ID":"faba1bc0-5c90-49c7-9fbf-b2a81a4d3516","Type":"ContainerStarted","Data":"7d95a96cbf664d23fb3fbfd0a891a65c3a9800714668fe6886eff28a2dd03ce9"}
Sep 17 03:20:45.312746 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:45.312697    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-bqnhq" podStartSLOduration=1.8751888079999999 podStartE2EDuration="3.312682971s" totalImagesPullingTime="1.437494163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:45.311312488 +0000 UTC m=+95.914408507" watchObservedRunningTime="2026-09-17 03:20:45.312682971 +0000 UTC m=+95.915778975"
Sep 17 03:20:47.171255 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.171221    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"]
Sep 17 03:20:47.174883 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.174865    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"
Sep 17 03:20:47.176838 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.176814    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 03:20:47.176838 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.176830    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 03:20:47.177015 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.176986    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gjcbq\""
Sep 17 03:20:47.184084 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.184066    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"]
Sep 17 03:20:47.192230 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.189238    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6ptrr"]
Sep 17 03:20:47.195523 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.195502    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.202151 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.197962    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"]
Sep 17 03:20:47.202151 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.198591    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-c2z72\""
Sep 17 03:20:47.202151 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.198649    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 03:20:47.202151 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.198815    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 03:20:47.202151 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.199007    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 03:20:47.202151 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.201370    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.203611 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.203592    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 03:20:47.203927 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.203904    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 03:20:47.204048 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.204034    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-thsv6\""
Sep 17 03:20:47.204457 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.204174    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 03:20:47.213658 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.213635    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"]
Sep 17 03:20:47.244417 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.244397    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:20:47.244417 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.244430    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:20:47.246782 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.246763    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls\") pod \"image-registry-c7659966b-292d6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") " pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:20:47.254401 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.254380    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cbfac4e-1716-45d1-8a32-139bccea48a9-metrics-tls\") pod \"dns-default-4fqdd\" (UID: \"7cbfac4e-1716-45d1-8a32-139bccea48a9\") " pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:20:47.345286 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345260    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s4h2\" (UniqueName: \"kubernetes.io/projected/a130fd36-fe3a-47e3-a437-078ff2f08676-kube-api-access-4s4h2\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.345390 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345294    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a130fd36-fe3a-47e3-a437-078ff2f08676-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.345390 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345310    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a130fd36-fe3a-47e3-a437-078ff2f08676-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.345390 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345328    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-textfile\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.345390 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345351    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a130fd36-fe3a-47e3-a437-078ff2f08676-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.345603 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345397    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a130fd36-fe3a-47e3-a437-078ff2f08676-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.345603 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345479    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a130fd36-fe3a-47e3-a437-078ff2f08676-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.345603 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345577    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a96ad582-ed86-424a-b88d-95f9ff7df8f6-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-vxqjh\" (UID: \"a96ad582-ed86-424a-b88d-95f9ff7df8f6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"
Sep 17 03:20:47.345603 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345598    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-tls\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.345804 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345614    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3da63e2a-f87a-4fc6-b710-0848c4b89488-metrics-client-ca\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.345804 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345639    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a96ad582-ed86-424a-b88d-95f9ff7df8f6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-vxqjh\" (UID: \"a96ad582-ed86-424a-b88d-95f9ff7df8f6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"
Sep 17 03:20:47.345804 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345656    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.345804 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345686    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.345804 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345733    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2rxm\" (UniqueName: \"kubernetes.io/projected/a96ad582-ed86-424a-b88d-95f9ff7df8f6-kube-api-access-t2rxm\") pod \"openshift-state-metrics-67dcf8c979-vxqjh\" (UID: \"a96ad582-ed86-424a-b88d-95f9ff7df8f6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"
Sep 17 03:20:47.345804 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345757    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3da63e2a-f87a-4fc6-b710-0848c4b89488-sys\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.345804 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345783    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a96ad582-ed86-424a-b88d-95f9ff7df8f6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-vxqjh\" (UID: \"a96ad582-ed86-424a-b88d-95f9ff7df8f6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"
Sep 17 03:20:47.346142 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3da63e2a-f87a-4fc6-b710-0848c4b89488-root\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.346142 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345843    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-wtmp\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.346142 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.345868    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p98kz\" (UniqueName: \"kubernetes.io/projected/3da63e2a-f87a-4fc6-b710-0848c4b89488-kube-api-access-p98kz\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.446572 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.446525    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a130fd36-fe3a-47e3-a437-078ff2f08676-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.446572 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.446550    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a130fd36-fe3a-47e3-a437-078ff2f08676-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.446572 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.446569    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-textfile\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.446781 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.446586    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a130fd36-fe3a-47e3-a437-078ff2f08676-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.446837 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.446803    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a130fd36-fe3a-47e3-a437-078ff2f08676-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.446889 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.446840    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a130fd36-fe3a-47e3-a437-078ff2f08676-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.446946 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.446894    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a96ad582-ed86-424a-b88d-95f9ff7df8f6-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-vxqjh\" (UID: \"a96ad582-ed86-424a-b88d-95f9ff7df8f6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"
Sep 17 03:20:47.446946 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.446907    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-textfile\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.446946 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.446925    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-tls\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.447096 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.446950    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3da63e2a-f87a-4fc6-b710-0848c4b89488-metrics-client-ca\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.447096 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.446971    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a130fd36-fe3a-47e3-a437-078ff2f08676-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.447096 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.446987    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a96ad582-ed86-424a-b88d-95f9ff7df8f6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-vxqjh\" (UID: \"a96ad582-ed86-424a-b88d-95f9ff7df8f6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"
Sep 17 03:20:47.447096 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.447014    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.447096 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.447058    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.447366 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.447096    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2rxm\" (UniqueName: \"kubernetes.io/projected/a96ad582-ed86-424a-b88d-95f9ff7df8f6-kube-api-access-t2rxm\") pod \"openshift-state-metrics-67dcf8c979-vxqjh\" (UID: \"a96ad582-ed86-424a-b88d-95f9ff7df8f6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"
Sep 17 03:20:47.447366 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.447122    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3da63e2a-f87a-4fc6-b710-0848c4b89488-sys\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.447366 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.447150    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a96ad582-ed86-424a-b88d-95f9ff7df8f6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-vxqjh\" (UID: \"a96ad582-ed86-424a-b88d-95f9ff7df8f6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"
Sep 17 03:20:47.447366 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.447174    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3da63e2a-f87a-4fc6-b710-0848c4b89488-root\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.447366 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.447324    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-wtmp\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.447366 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.447350    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p98kz\" (UniqueName: \"kubernetes.io/projected/3da63e2a-f87a-4fc6-b710-0848c4b89488-kube-api-access-p98kz\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.447653 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.447400    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s4h2\" (UniqueName: \"kubernetes.io/projected/a130fd36-fe3a-47e3-a437-078ff2f08676-kube-api-access-4s4h2\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.447653 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.447415    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a130fd36-fe3a-47e3-a437-078ff2f08676-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.447653 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.447545    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a130fd36-fe3a-47e3-a437-078ff2f08676-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.448513 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.447862    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3da63e2a-f87a-4fc6-b710-0848c4b89488-root\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.448513 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.447935    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3da63e2a-f87a-4fc6-b710-0848c4b89488-metrics-client-ca\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.448513 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:47.448021    2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 03:20:47.448513 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:47.448074    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-tls podName:3da63e2a-f87a-4fc6-b710-0848c4b89488 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:47.948056682 +0000 UTC m=+98.551152668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-tls") pod "node-exporter-6ptrr" (UID: "3da63e2a-f87a-4fc6-b710-0848c4b89488") : secret "node-exporter-tls" not found
Sep 17 03:20:47.448513 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.448362    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3da63e2a-f87a-4fc6-b710-0848c4b89488-sys\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.448513 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.448395    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-accelerators-collector-config\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.448867 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.448535    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-wtmp\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.449371 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.449338    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a96ad582-ed86-424a-b88d-95f9ff7df8f6-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-vxqjh\" (UID: \"a96ad582-ed86-424a-b88d-95f9ff7df8f6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"
Sep 17 03:20:47.449575 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.449462    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a130fd36-fe3a-47e3-a437-078ff2f08676-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.449971 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.449946    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a130fd36-fe3a-47e3-a437-078ff2f08676-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.450369 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.450343    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a96ad582-ed86-424a-b88d-95f9ff7df8f6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-vxqjh\" (UID: \"a96ad582-ed86-424a-b88d-95f9ff7df8f6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"
Sep 17 03:20:47.450755 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.450719    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.451154 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.451133    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a96ad582-ed86-424a-b88d-95f9ff7df8f6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-vxqjh\" (UID: \"a96ad582-ed86-424a-b88d-95f9ff7df8f6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"
Sep 17 03:20:47.455925 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.455900    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p98kz\" (UniqueName: \"kubernetes.io/projected/3da63e2a-f87a-4fc6-b710-0848c4b89488-kube-api-access-p98kz\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.456128 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.456108    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2rxm\" (UniqueName: \"kubernetes.io/projected/a96ad582-ed86-424a-b88d-95f9ff7df8f6-kube-api-access-t2rxm\") pod \"openshift-state-metrics-67dcf8c979-vxqjh\" (UID: \"a96ad582-ed86-424a-b88d-95f9ff7df8f6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"
Sep 17 03:20:47.456250 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.456232    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s4h2\" (UniqueName: \"kubernetes.io/projected/a130fd36-fe3a-47e3-a437-078ff2f08676-kube-api-access-4s4h2\") pod \"kube-state-metrics-59fbdd77d-rjm62\" (UID: \"a130fd36-fe3a-47e3-a437-078ff2f08676\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.486166 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.486146    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"
Sep 17 03:20:47.505367 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.505343    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lh58r\""
Sep 17 03:20:47.514320 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.514119    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:20:47.515386 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.515360    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"
Sep 17 03:20:47.515822 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.515804    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8xsvg\""
Sep 17 03:20:47.525476 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.525270    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:20:47.781217 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.781057    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh"]
Sep 17 03:20:47.783713 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:47.783692    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96ad582_ed86_424a_b88d_95f9ff7df8f6.slice/crio-84d292ae41ee1ad8ab3ed418e8f243509dba8591af0b349eac4f71dba249a7d0 WatchSource:0}: Error finding container 84d292ae41ee1ad8ab3ed418e8f243509dba8591af0b349eac4f71dba249a7d0: Status 404 returned error can't find the container with id 84d292ae41ee1ad8ab3ed418e8f243509dba8591af0b349eac4f71dba249a7d0
Sep 17 03:20:47.813290 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.813267    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62"]
Sep 17 03:20:47.814977 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:47.814938    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda130fd36_fe3a_47e3_a437_078ff2f08676.slice/crio-6e08710f2faa64b2c881253ed8ea528081902dd4b3b85fbc90c675ea093c388a WatchSource:0}: Error finding container 6e08710f2faa64b2c881253ed8ea528081902dd4b3b85fbc90c675ea093c388a: Status 404 returned error can't find the container with id 6e08710f2faa64b2c881253ed8ea528081902dd4b3b85fbc90c675ea093c388a
Sep 17 03:20:47.816039 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.815951    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c7659966b-292d6"]
Sep 17 03:20:47.818915 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:47.818897    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88b24fe_fd7e_4c77_8648_423434670be6.slice/crio-7b8aa995fbc65805a4315ecb1f5eccafb1489c559d387042320353fd2f7b9c50 WatchSource:0}: Error finding container 7b8aa995fbc65805a4315ecb1f5eccafb1489c559d387042320353fd2f7b9c50: Status 404 returned error can't find the container with id 7b8aa995fbc65805a4315ecb1f5eccafb1489c559d387042320353fd2f7b9c50
Sep 17 03:20:47.849074 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.849047    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4fqdd"]
Sep 17 03:20:47.856108 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:47.856085    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbfac4e_1716_45d1_8a32_139bccea48a9.slice/crio-de0399ab95ec491de1709b5105d39640943dd97cef13ae7c1434a64cc30319c7 WatchSource:0}: Error finding container de0399ab95ec491de1709b5105d39640943dd97cef13ae7c1434a64cc30319c7: Status 404 returned error can't find the container with id de0399ab95ec491de1709b5105d39640943dd97cef13ae7c1434a64cc30319c7
Sep 17 03:20:47.952521 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:47.952490    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-tls\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:47.952686 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:47.952664    2557 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 03:20:47.952753 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:47.952741    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-tls podName:3da63e2a-f87a-4fc6-b710-0848c4b89488 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:48.952721601 +0000 UTC m=+99.555817607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-tls") pod "node-exporter-6ptrr" (UID: "3da63e2a-f87a-4fc6-b710-0848c4b89488") : secret "node-exporter-tls" not found
Sep 17 03:20:48.309807 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:48.309766    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4fqdd" event={"ID":"7cbfac4e-1716-45d1-8a32-139bccea48a9","Type":"ContainerStarted","Data":"de0399ab95ec491de1709b5105d39640943dd97cef13ae7c1434a64cc30319c7"}
Sep 17 03:20:48.311327 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:48.311279    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62" event={"ID":"a130fd36-fe3a-47e3-a437-078ff2f08676","Type":"ContainerStarted","Data":"6e08710f2faa64b2c881253ed8ea528081902dd4b3b85fbc90c675ea093c388a"}
Sep 17 03:20:48.313621 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:48.313597    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh" event={"ID":"a96ad582-ed86-424a-b88d-95f9ff7df8f6","Type":"ContainerStarted","Data":"b428752104d00d54d350f7c8165b0be5cc9c6810c3baaffb6479855d0f236abb"}
Sep 17 03:20:48.313726 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:48.313654    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh" event={"ID":"a96ad582-ed86-424a-b88d-95f9ff7df8f6","Type":"ContainerStarted","Data":"1e0ed3f92304dd6cc39b9003e74e529e8ab35008d302c8f7d3d2b6f62c4af03f"}
Sep 17 03:20:48.313726 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:48.313682    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh" event={"ID":"a96ad582-ed86-424a-b88d-95f9ff7df8f6","Type":"ContainerStarted","Data":"84d292ae41ee1ad8ab3ed418e8f243509dba8591af0b349eac4f71dba249a7d0"}
Sep 17 03:20:48.315680 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:48.315606    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c7659966b-292d6" event={"ID":"f88b24fe-fd7e-4c77-8648-423434670be6","Type":"ContainerStarted","Data":"3622a39dd6f9d77367e5fddf40cbc006bb584d664b04601b0cd315d7ac9c984c"}
Sep 17 03:20:48.315680 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:48.315646    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c7659966b-292d6" event={"ID":"f88b24fe-fd7e-4c77-8648-423434670be6","Type":"ContainerStarted","Data":"7b8aa995fbc65805a4315ecb1f5eccafb1489c559d387042320353fd2f7b9c50"}
Sep 17 03:20:48.315854 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:48.315769    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:20:48.341132 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:48.341083    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c7659966b-292d6" podStartSLOduration=98.341062704 podStartE2EDuration="1m38.341062704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:48.339522957 +0000 UTC m=+98.942618962" watchObservedRunningTime="2026-09-17 03:20:48.341062704 +0000 UTC m=+98.944158711"
Sep 17 03:20:48.908805 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:48.908772    2557 scope.go:117] "RemoveContainer" containerID="95cb6c8d2cf474ee4db604f2f09cf338c1341f2d81be96b317dee8d80c08f4dd"
Sep 17 03:20:48.908984 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:48.908959    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-qvxrt_openshift-console-operator(ba4e66fe-1155-4f15-841f-b5eadceec705)\"" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt" podUID="ba4e66fe-1155-4f15-841f-b5eadceec705"
Sep 17 03:20:48.962122 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:48.962086    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-tls\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:48.964899 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:48.964849    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3da63e2a-f87a-4fc6-b710-0848c4b89488-node-exporter-tls\") pod \"node-exporter-6ptrr\" (UID: \"3da63e2a-f87a-4fc6-b710-0848c4b89488\") " pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:49.008333 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:49.008303    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6ptrr"
Sep 17 03:20:49.936266 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:49.936242    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da63e2a_f87a_4fc6_b710_0848c4b89488.slice/crio-3cdcf5e5e41d32d5a1bc2236ddd98da0d636d18115579493f0821d0be3526acd WatchSource:0}: Error finding container 3cdcf5e5e41d32d5a1bc2236ddd98da0d636d18115579493f0821d0be3526acd: Status 404 returned error can't find the container with id 3cdcf5e5e41d32d5a1bc2236ddd98da0d636d18115579493f0821d0be3526acd
Sep 17 03:20:50.333059 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:50.332442    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62" event={"ID":"a130fd36-fe3a-47e3-a437-078ff2f08676","Type":"ContainerStarted","Data":"715d54fbd3708481bac597cf026d497c041e15c31791c1ebce77f1c75f71b714"}
Sep 17 03:20:50.333059 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:50.332510    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62" event={"ID":"a130fd36-fe3a-47e3-a437-078ff2f08676","Type":"ContainerStarted","Data":"e31e1fd2fc95e0dc7a614f40b21516dc06498fa7807a34f705869fedc6a95d6f"}
Sep 17 03:20:50.333059 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:50.332537    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62" event={"ID":"a130fd36-fe3a-47e3-a437-078ff2f08676","Type":"ContainerStarted","Data":"87e8f0e937b6c441cceb2b39e5da65e2f85f35d262d07978e21a99bf3b750966"}
Sep 17 03:20:50.336598 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:50.335930    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh" event={"ID":"a96ad582-ed86-424a-b88d-95f9ff7df8f6","Type":"ContainerStarted","Data":"32b7062abe8252340d93496ad7e385ca7669e551bb64b41cd839251dcca258ff"}
Sep 17 03:20:50.337492 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:50.337438    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ptrr" event={"ID":"3da63e2a-f87a-4fc6-b710-0848c4b89488","Type":"ContainerStarted","Data":"3cdcf5e5e41d32d5a1bc2236ddd98da0d636d18115579493f0821d0be3526acd"}
Sep 17 03:20:50.340524 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:50.340403    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4fqdd" event={"ID":"7cbfac4e-1716-45d1-8a32-139bccea48a9","Type":"ContainerStarted","Data":"b3311886913e014abb60c59d20000783debfb921bd7616048c1f5543e4ea7b24"}
Sep 17 03:20:50.340524 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:50.340446    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4fqdd" event={"ID":"7cbfac4e-1716-45d1-8a32-139bccea48a9","Type":"ContainerStarted","Data":"c90e50fc1cf4d3533936d161c69f2b37cc0f13c0b470ca9e9a08c86bf93f9a23"}
Sep 17 03:20:50.340695 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:50.340562    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:20:50.353350 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:50.353303    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rjm62" podStartSLOduration=1.236701472 podStartE2EDuration="3.353285548s" totalImagesPullingTime="2.116584076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:50.351776115 +0000 UTC m=+100.954872144" watchObservedRunningTime="2026-09-17 03:20:50.353285548 +0000 UTC m=+100.956381553"
Sep 17 03:20:50.373106 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:50.372740    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-vxqjh" podStartSLOduration=1.300046203 podStartE2EDuration="3.372720764s" totalImagesPullingTime="2.072674561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:50.371480203 +0000 UTC m=+100.974576208" watchObservedRunningTime="2026-09-17 03:20:50.372720764 +0000 UTC m=+100.975816766"
Sep 17 03:20:50.393380 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:50.393310    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4fqdd" podStartSLOduration=65.317711661 podStartE2EDuration="1m7.393294716s" totalImagesPullingTime="2.075583055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:50.392489275 +0000 UTC m=+100.995585280" watchObservedRunningTime="2026-09-17 03:20:50.393294716 +0000 UTC m=+100.996390721"
Sep 17 03:20:51.352082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.352038    2557 generic.go:356] "Generic (PLEG): container finished" podID="3da63e2a-f87a-4fc6-b710-0848c4b89488" containerID="2d978c33ca9dffda525c50dc3d3356a84bd5807a52c66108d44cdb61da4aea28" exitCode=0
Sep 17 03:20:51.352540 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.352149    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ptrr" event={"ID":"3da63e2a-f87a-4fc6-b710-0848c4b89488","Type":"ContainerDied","Data":"2d978c33ca9dffda525c50dc3d3356a84bd5807a52c66108d44cdb61da4aea28"}
Sep 17 03:20:51.601099 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.601067    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-9b7d74d76-fnw2t"]
Sep 17 03:20:51.604219 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.604166    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.606623 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.606600    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 03:20:51.606728 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.606669    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-89tdhu7j0qg9s\""
Sep 17 03:20:51.606905 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.606890    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 03:20:51.606965 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.606942    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 03:20:51.606965 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.606957    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-95654\""
Sep 17 03:20:51.607064 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.606967    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 03:20:51.615001 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.614978    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9b7d74d76-fnw2t"]
Sep 17 03:20:51.688682 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.688657    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-client-ca-bundle\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.688773 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.688693    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.688773 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.688713    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wbhr\" (UniqueName: \"kubernetes.io/projected/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-kube-api-access-9wbhr\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.688841 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.688780    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-metrics-server-audit-profiles\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.688841 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.688824    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-secret-metrics-server-tls\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.688905 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.688848    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-audit-log\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.688905 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.688879    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-secret-metrics-server-client-certs\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.790038 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.790018    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-secret-metrics-server-tls\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.790142 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.790049    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-audit-log\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.790142 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.790079    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-secret-metrics-server-client-certs\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.790142 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.790125    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-client-ca-bundle\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.790309 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.790164    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.790309 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.790214    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wbhr\" (UniqueName: \"kubernetes.io/projected/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-kube-api-access-9wbhr\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.790309 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.790253    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-metrics-server-audit-profiles\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.790626 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.790595    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-audit-log\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.791038 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.791012    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.791164 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.791145    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-metrics-server-audit-profiles\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.792669 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.792644    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-secret-metrics-server-tls\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.792916 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.792896    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-secret-metrics-server-client-certs\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.792957 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.792911    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-client-ca-bundle\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.797369 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.797350    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wbhr\" (UniqueName: \"kubernetes.io/projected/70e857a5-6ddf-4811-9b58-8ff90d0e27cc-kube-api-access-9wbhr\") pod \"metrics-server-9b7d74d76-fnw2t\" (UID: \"70e857a5-6ddf-4811-9b58-8ff90d0e27cc\") " pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.912883 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.912657    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:20:51.950502 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.950474    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq"]
Sep 17 03:20:51.954831 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.954808    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq"
Sep 17 03:20:51.956912 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.956890    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 03:20:51.957033 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.956897    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6wjfm\""
Sep 17 03:20:51.962684 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:51.962286    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq"]
Sep 17 03:20:52.092809 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:52.092778    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlzzn\" (UniqueName: \"kubernetes.io/projected/48f39267-6e56-4d92-9ce0-c14b039ebad9-kube-api-access-dlzzn\") pod \"monitoring-plugin-5d85778b9d-mw5fq\" (UID: \"48f39267-6e56-4d92-9ce0-c14b039ebad9\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq"
Sep 17 03:20:52.092958 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:52.092872    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48f39267-6e56-4d92-9ce0-c14b039ebad9-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-mw5fq\" (UID: \"48f39267-6e56-4d92-9ce0-c14b039ebad9\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq"
Sep 17 03:20:52.194138 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:52.194077    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48f39267-6e56-4d92-9ce0-c14b039ebad9-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-mw5fq\" (UID: \"48f39267-6e56-4d92-9ce0-c14b039ebad9\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq"
Sep 17 03:20:52.194138 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:52.194118    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlzzn\" (UniqueName: \"kubernetes.io/projected/48f39267-6e56-4d92-9ce0-c14b039ebad9-kube-api-access-dlzzn\") pod \"monitoring-plugin-5d85778b9d-mw5fq\" (UID: \"48f39267-6e56-4d92-9ce0-c14b039ebad9\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq"
Sep 17 03:20:52.194307 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:52.194226    2557 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 03:20:52.194307 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:20:52.194293    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f39267-6e56-4d92-9ce0-c14b039ebad9-monitoring-plugin-cert podName:48f39267-6e56-4d92-9ce0-c14b039ebad9 nodeName:}" failed. No retries permitted until 2026-09-17 03:20:52.694279411 +0000 UTC m=+103.297375397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/48f39267-6e56-4d92-9ce0-c14b039ebad9-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-mw5fq" (UID: "48f39267-6e56-4d92-9ce0-c14b039ebad9") : secret "monitoring-plugin-cert" not found
Sep 17 03:20:52.199852 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:52.199830    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n5bhs"
Sep 17 03:20:52.222244 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:52.222164    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9b7d74d76-fnw2t"]
Sep 17 03:20:52.222998 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:52.222976    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlzzn\" (UniqueName: \"kubernetes.io/projected/48f39267-6e56-4d92-9ce0-c14b039ebad9-kube-api-access-dlzzn\") pod \"monitoring-plugin-5d85778b9d-mw5fq\" (UID: \"48f39267-6e56-4d92-9ce0-c14b039ebad9\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq"
Sep 17 03:20:52.224500 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:52.224482    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e857a5_6ddf_4811_9b58_8ff90d0e27cc.slice/crio-7f7ca3864609cead6b18b24dc14286e396c62a9b842fe301e3e369f0a9ec6770 WatchSource:0}: Error finding container 7f7ca3864609cead6b18b24dc14286e396c62a9b842fe301e3e369f0a9ec6770: Status 404 returned error can't find the container with id 7f7ca3864609cead6b18b24dc14286e396c62a9b842fe301e3e369f0a9ec6770
Sep 17 03:20:52.360959 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:52.359368    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ptrr" event={"ID":"3da63e2a-f87a-4fc6-b710-0848c4b89488","Type":"ContainerStarted","Data":"40c87abdfa76f1b56ab0eb45b9d84030356dfcebf27f97ba0cc204f5a2037482"}
Sep 17 03:20:52.360959 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:52.359425    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6ptrr" event={"ID":"3da63e2a-f87a-4fc6-b710-0848c4b89488","Type":"ContainerStarted","Data":"3cf84edfbd1642f0796c1412fdff8d23a44b3792c76d8db657255ac7532dce2c"}
Sep 17 03:20:52.362413 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:52.362383    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t" event={"ID":"70e857a5-6ddf-4811-9b58-8ff90d0e27cc","Type":"ContainerStarted","Data":"7f7ca3864609cead6b18b24dc14286e396c62a9b842fe301e3e369f0a9ec6770"}
Sep 17 03:20:52.398016 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:52.397970    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6ptrr" podStartSLOduration=4.420185525 podStartE2EDuration="5.397952358s" totalImagesPullingTime="917.73594ms" totalInitContainerRuntime="60.030893ms" isStatefulPod=true podCreationTimestamp="2026-09-17 03:20:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:52.397477687 +0000 UTC m=+103.000573704" watchObservedRunningTime="2026-09-17 03:20:52.397952358 +0000 UTC m=+103.001048364"
Sep 17 03:20:52.699380 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:52.699355    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48f39267-6e56-4d92-9ce0-c14b039ebad9-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-mw5fq\" (UID: \"48f39267-6e56-4d92-9ce0-c14b039ebad9\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq"
Sep 17 03:20:52.701610 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:52.701587    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48f39267-6e56-4d92-9ce0-c14b039ebad9-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-mw5fq\" (UID: \"48f39267-6e56-4d92-9ce0-c14b039ebad9\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq"
Sep 17 03:20:52.865831 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:52.865806    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq"
Sep 17 03:20:53.142861 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.141740    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq"]
Sep 17 03:20:53.147371 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:53.147299    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f39267_6e56_4d92_9ce0_c14b039ebad9.slice/crio-1abe8b666b6729c9ea5beedce276b296412d7c876cfe7e9e160b6ec1cf022d29 WatchSource:0}: Error finding container 1abe8b666b6729c9ea5beedce276b296412d7c876cfe7e9e160b6ec1cf022d29: Status 404 returned error can't find the container with id 1abe8b666b6729c9ea5beedce276b296412d7c876cfe7e9e160b6ec1cf022d29
Sep 17 03:20:53.368584 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.368546    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq" event={"ID":"48f39267-6e56-4d92-9ce0-c14b039ebad9","Type":"ContainerStarted","Data":"1abe8b666b6729c9ea5beedce276b296412d7c876cfe7e9e160b6ec1cf022d29"}
Sep 17 03:20:53.490055 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.489958    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 03:20:53.495360 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.495336    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.498153 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.498128    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 03:20:53.498296 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.498257    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 03:20:53.498518 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.498494    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 03:20:53.498621 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.498524    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 03:20:53.498621 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.498548    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 03:20:53.498749 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.498687    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 03:20:53.498825 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.498770    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 03:20:53.498825 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.498789    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 03:20:53.499123 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.499101    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r2jjr\""
Sep 17 03:20:53.499540 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.499517    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 03:20:53.504422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.502119    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 03:20:53.504422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.502353    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 03:20:53.504422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.502623    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-evli7s2s64m73\""
Sep 17 03:20:53.504422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.503206    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 03:20:53.505393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.505368    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 03:20:53.510147 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.510123    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 03:20:53.514220 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.514183    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 03:20:53.608583 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.608554    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.608583 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.608588    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.608822 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.608616    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.608822 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.608691    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-web-config\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.608822 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.608732    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.608822 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.608755    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.608973 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.608818    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6beac4c9-90c9-42a2-8dac-15062d2a63f2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.608973 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.608863    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.608973 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.608908    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.608973 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.608934    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.609125 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.608980    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.609125 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.609010    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs4b6\" (UniqueName: \"kubernetes.io/projected/6beac4c9-90c9-42a2-8dac-15062d2a63f2-kube-api-access-qs4b6\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.609125 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.609032    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.609125 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.609068    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.609125 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.609100    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.609422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.609137    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.609422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.609265    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6beac4c9-90c9-42a2-8dac-15062d2a63f2-config-out\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.609422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.609300    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.609422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.609331    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-config\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.609422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.609360    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.710837 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.710803    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.710990 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.710847    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711052 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711015    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6beac4c9-90c9-42a2-8dac-15062d2a63f2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711102 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711059    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711155 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711100    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711155 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711129    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711291 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711155    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711291 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711180    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs4b6\" (UniqueName: \"kubernetes.io/projected/6beac4c9-90c9-42a2-8dac-15062d2a63f2-kube-api-access-qs4b6\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711291 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711221    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711291 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711263    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711482 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711290    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711482 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711331    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711482 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711378    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6beac4c9-90c9-42a2-8dac-15062d2a63f2-config-out\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711482 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711405    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711482 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711434    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-config\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711482 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711456    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711783 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711519    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711783 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711542    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711783 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711583    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711783 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711611    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-web-config\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711977 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711829    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.711977 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.711920    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.712401 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.712366    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.713156 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.713072    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.715314 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.715289    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.716448 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.715746    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6beac4c9-90c9-42a2-8dac-15062d2a63f2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.716448 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.716270    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.716448 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.716346    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.716448 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.716398    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.716827 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.716805    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-web-config\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.717530 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.717510    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.717776 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.717739    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.718452 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.718430    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-config\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.718849 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.718826    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.719049 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.719002    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6beac4c9-90c9-42a2-8dac-15062d2a63f2-config-out\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.719958 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.719893    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.720059 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.719967    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.721710 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.721668    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.722102 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.722076    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.734540 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.734521    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs4b6\" (UniqueName: \"kubernetes.io/projected/6beac4c9-90c9-42a2-8dac-15062d2a63f2-kube-api-access-qs4b6\") pod \"prometheus-k8s-0\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:53.810367 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:53.809999    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:20:54.251208 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:54.251155    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 03:20:54.254499 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:20:54.254472    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6beac4c9_90c9_42a2_8dac_15062d2a63f2.slice/crio-a2b882225f21693b115fd8bd5df8b0721dd8072cfc49161524f0a43ff1e3a5d7 WatchSource:0}: Error finding container a2b882225f21693b115fd8bd5df8b0721dd8072cfc49161524f0a43ff1e3a5d7: Status 404 returned error can't find the container with id a2b882225f21693b115fd8bd5df8b0721dd8072cfc49161524f0a43ff1e3a5d7
Sep 17 03:20:54.374798 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:54.374755    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6beac4c9-90c9-42a2-8dac-15062d2a63f2","Type":"ContainerStarted","Data":"a2b882225f21693b115fd8bd5df8b0721dd8072cfc49161524f0a43ff1e3a5d7"}
Sep 17 03:20:54.376591 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:54.376559    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t" event={"ID":"70e857a5-6ddf-4811-9b58-8ff90d0e27cc","Type":"ContainerStarted","Data":"b44387168efff778dca5809c2377b9e2cdf172afbe5291bea09761d34f76c4e6"}
Sep 17 03:20:54.394724 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:54.394678    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t" podStartSLOduration=1.596779103 podStartE2EDuration="3.394664537s" totalImagesPullingTime="1.797885434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:54.394212857 +0000 UTC m=+104.997308864" watchObservedRunningTime="2026-09-17 03:20:54.394664537 +0000 UTC m=+104.997760541"
Sep 17 03:20:56.387230 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:56.387180    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq" event={"ID":"48f39267-6e56-4d92-9ce0-c14b039ebad9","Type":"ContainerStarted","Data":"fb5c1377c547184e5e291689e114908160baaf4f2a06908a621f26659947ada0"}
Sep 17 03:20:56.387581 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:56.387342    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq"
Sep 17 03:20:56.389040 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:56.389010    2557 generic.go:356] "Generic (PLEG): container finished" podID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerID="09edf0d5e96db2aee6085a377759c42f06976cbfdcf8cf79239246a5ab4bd226" exitCode=0
Sep 17 03:20:56.389123 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:56.389051    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6beac4c9-90c9-42a2-8dac-15062d2a63f2","Type":"ContainerDied","Data":"09edf0d5e96db2aee6085a377759c42f06976cbfdcf8cf79239246a5ab4bd226"}
Sep 17 03:20:56.393256 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:56.393237    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq"
Sep 17 03:20:56.401720 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:56.401679    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-mw5fq" podStartSLOduration=2.804536395 podStartE2EDuration="5.401666355s" totalImagesPullingTime="2.59712996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:20:56.401171577 +0000 UTC m=+107.004267580" watchObservedRunningTime="2026-09-17 03:20:56.401666355 +0000 UTC m=+107.004762359"
Sep 17 03:20:58.509040 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:58.509011    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c7659966b-292d6"]
Sep 17 03:20:59.910737 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:20:59.910711    2557 scope.go:117] "RemoveContainer" containerID="95cb6c8d2cf474ee4db604f2f09cf338c1341f2d81be96b317dee8d80c08f4dd"
Sep 17 03:21:00.354396 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:00.354369    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4fqdd"
Sep 17 03:21:00.410933 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:00.410904    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:21:00.411102 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:00.411017    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt" event={"ID":"ba4e66fe-1155-4f15-841f-b5eadceec705","Type":"ContainerStarted","Data":"2e5e75dc46fcf230b4bae911cced531e0796cc230471b3a01a7c95ef48b78db8"}
Sep 17 03:21:00.411385 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:00.411361    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:21:00.413506 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:00.413483    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6beac4c9-90c9-42a2-8dac-15062d2a63f2","Type":"ContainerStarted","Data":"2c5d9612838d74a476ffd5ad8e8b47e401fad1064a560a7e4487dbf6af309d2a"}
Sep 17 03:21:00.413614 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:00.413581    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6beac4c9-90c9-42a2-8dac-15062d2a63f2","Type":"ContainerStarted","Data":"075c52ea3be9bd0d38cf9b69f4f43f8b2f32c33f7a16bb087c28479d3e2b3383"}
Sep 17 03:21:00.596636 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:00.596611    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-qvxrt"
Sep 17 03:21:02.425773 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:02.425737    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6beac4c9-90c9-42a2-8dac-15062d2a63f2","Type":"ContainerStarted","Data":"4f361545becd4a800f14e81e175e3e158b3e802572bdb37d28984d3953d50a94"}
Sep 17 03:21:02.426126 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:02.425808    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6beac4c9-90c9-42a2-8dac-15062d2a63f2","Type":"ContainerStarted","Data":"1e4c6d7eba065950e39c632c996245f663589840c9873ad3411d5844961256d1"}
Sep 17 03:21:02.426126 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:02.425836    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6beac4c9-90c9-42a2-8dac-15062d2a63f2","Type":"ContainerStarted","Data":"f9e2a0f0b02e60f8768db1eb9d40eccadeaffb66111987312db32c3f85f2826f"}
Sep 17 03:21:02.426126 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:02.425861    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6beac4c9-90c9-42a2-8dac-15062d2a63f2","Type":"ContainerStarted","Data":"9c383734cf9143552c30097adfd02784d86b16fd1b8847debec5bb6e00ad04f7"}
Sep 17 03:21:02.456486 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:02.456417    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.327748072 podStartE2EDuration="9.456386156s" totalImagesPullingTime="7.11401771s" totalInitContainerRuntime="14.620374ms" isStatefulPod=false podCreationTimestamp="2026-09-17 03:20:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:21:02.45488664 +0000 UTC m=+113.057982647" watchObservedRunningTime="2026-09-17 03:21:02.456386156 +0000 UTC m=+113.059482163"
Sep 17 03:21:03.811402 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:03.811356    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:21:08.514482 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:08.514451    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:21:11.913255 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:11.913225    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:21:11.913697 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:11.913441    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:21:19.719974 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:19.719934    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs\") pod \"network-metrics-daemon-5d8wb\" (UID: \"c7146f5b-e9f5-4469-8161-756c03f69dac\") " pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:21:19.722628 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:19.722593    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7146f5b-e9f5-4469-8161-756c03f69dac-metrics-certs\") pod \"network-metrics-daemon-5d8wb\" (UID: \"c7146f5b-e9f5-4469-8161-756c03f69dac\") " pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:21:19.939857 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:19.939827    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tmrnq\""
Sep 17 03:21:19.948479 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:19.948452    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5d8wb"
Sep 17 03:21:20.217574 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:20.217545    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5d8wb"]
Sep 17 03:21:20.220576 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:21:20.220552    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7146f5b_e9f5_4469_8161_756c03f69dac.slice/crio-c7f137bdb141af7a18104ff706cdacb273c78c9a70f58ddf9b3d0e2f7b24324d WatchSource:0}: Error finding container c7f137bdb141af7a18104ff706cdacb273c78c9a70f58ddf9b3d0e2f7b24324d: Status 404 returned error can't find the container with id c7f137bdb141af7a18104ff706cdacb273c78c9a70f58ddf9b3d0e2f7b24324d
Sep 17 03:21:20.500419 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:20.500374    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5d8wb" event={"ID":"c7146f5b-e9f5-4469-8161-756c03f69dac","Type":"ContainerStarted","Data":"c7f137bdb141af7a18104ff706cdacb273c78c9a70f58ddf9b3d0e2f7b24324d"}
Sep 17 03:21:22.510478 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:22.510444    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5d8wb" event={"ID":"c7146f5b-e9f5-4469-8161-756c03f69dac","Type":"ContainerStarted","Data":"f908dcd6eb59d17b4cb62f1b88c2a62eeb204b432a82d2cd762f00edd21711ac"}
Sep 17 03:21:22.510841 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:22.510502    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5d8wb" event={"ID":"c7146f5b-e9f5-4469-8161-756c03f69dac","Type":"ContainerStarted","Data":"928c4709196ab84daea74f21eb1a03fbc8493b08745db71e2cd24012d36bd0b8"}
Sep 17 03:21:22.526026 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:22.525972    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5d8wb" podStartSLOduration=132.157525114 podStartE2EDuration="2m13.525957015s" totalImagesPullingTime="1.368431901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:21:22.524242561 +0000 UTC m=+133.127338565" watchObservedRunningTime="2026-09-17 03:21:22.525957015 +0000 UTC m=+133.129053018"
Sep 17 03:21:23.533073 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.533035    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-c7659966b-292d6" podUID="f88b24fe-fd7e-4c77-8648-423434670be6" containerName="registry" containerID="cri-o://3622a39dd6f9d77367e5fddf40cbc006bb584d664b04601b0cd315d7ac9c984c" gracePeriod=30
Sep 17 03:21:23.781727 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.781703    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:21:23.853759 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.853696    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f88b24fe-fd7e-4c77-8648-423434670be6-image-registry-private-configuration\") pod \"f88b24fe-fd7e-4c77-8648-423434670be6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") "
Sep 17 03:21:23.853759 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.853748    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls\") pod \"f88b24fe-fd7e-4c77-8648-423434670be6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") "
Sep 17 03:21:23.853935 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.853795    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f88b24fe-fd7e-4c77-8648-423434670be6-registry-certificates\") pod \"f88b24fe-fd7e-4c77-8648-423434670be6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") "
Sep 17 03:21:23.853935 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.853826    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f88b24fe-fd7e-4c77-8648-423434670be6-ca-trust-extracted\") pod \"f88b24fe-fd7e-4c77-8648-423434670be6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") "
Sep 17 03:21:23.854043 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.853962    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f88b24fe-fd7e-4c77-8648-423434670be6-trusted-ca\") pod \"f88b24fe-fd7e-4c77-8648-423434670be6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") "
Sep 17 03:21:23.854043 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.854009    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-bound-sa-token\") pod \"f88b24fe-fd7e-4c77-8648-423434670be6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") "
Sep 17 03:21:23.854043 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.854039    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f88b24fe-fd7e-4c77-8648-423434670be6-installation-pull-secrets\") pod \"f88b24fe-fd7e-4c77-8648-423434670be6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") "
Sep 17 03:21:23.854188 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.854063    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ggvd\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-kube-api-access-6ggvd\") pod \"f88b24fe-fd7e-4c77-8648-423434670be6\" (UID: \"f88b24fe-fd7e-4c77-8648-423434670be6\") "
Sep 17 03:21:23.854298 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.854271    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88b24fe-fd7e-4c77-8648-423434670be6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f88b24fe-fd7e-4c77-8648-423434670be6" (UID: "f88b24fe-fd7e-4c77-8648-423434670be6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:21:23.854414 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.854388    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88b24fe-fd7e-4c77-8648-423434670be6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f88b24fe-fd7e-4c77-8648-423434670be6" (UID: "f88b24fe-fd7e-4c77-8648-423434670be6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:21:23.856533 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.856495    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f88b24fe-fd7e-4c77-8648-423434670be6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f88b24fe-fd7e-4c77-8648-423434670be6" (UID: "f88b24fe-fd7e-4c77-8648-423434670be6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:21:23.856680 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.856654    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f88b24fe-fd7e-4c77-8648-423434670be6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f88b24fe-fd7e-4c77-8648-423434670be6" (UID: "f88b24fe-fd7e-4c77-8648-423434670be6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:21:23.856738 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.856691    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f88b24fe-fd7e-4c77-8648-423434670be6" (UID: "f88b24fe-fd7e-4c77-8648-423434670be6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:21:23.856825 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.856811    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-kube-api-access-6ggvd" (OuterVolumeSpecName: "kube-api-access-6ggvd") pod "f88b24fe-fd7e-4c77-8648-423434670be6" (UID: "f88b24fe-fd7e-4c77-8648-423434670be6"). InnerVolumeSpecName "kube-api-access-6ggvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:21:23.856941 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.856927    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f88b24fe-fd7e-4c77-8648-423434670be6" (UID: "f88b24fe-fd7e-4c77-8648-423434670be6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:21:23.862224 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.862169    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f88b24fe-fd7e-4c77-8648-423434670be6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f88b24fe-fd7e-4c77-8648-423434670be6" (UID: "f88b24fe-fd7e-4c77-8648-423434670be6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:21:23.954939 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.954915    2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-registry-tls\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:21:23.954939 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.954937    2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f88b24fe-fd7e-4c77-8648-423434670be6-registry-certificates\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:21:23.955049 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.954947    2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f88b24fe-fd7e-4c77-8648-423434670be6-ca-trust-extracted\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:21:23.955049 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.954955    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f88b24fe-fd7e-4c77-8648-423434670be6-trusted-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:21:23.955049 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.954964    2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-bound-sa-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:21:23.955049 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.954971    2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f88b24fe-fd7e-4c77-8648-423434670be6-installation-pull-secrets\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:21:23.955049 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.954982    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ggvd\" (UniqueName: \"kubernetes.io/projected/f88b24fe-fd7e-4c77-8648-423434670be6-kube-api-access-6ggvd\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:21:23.955049 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:23.954996    2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f88b24fe-fd7e-4c77-8648-423434670be6-image-registry-private-configuration\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:21:24.520625 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:24.520597    2557 generic.go:356] "Generic (PLEG): container finished" podID="f88b24fe-fd7e-4c77-8648-423434670be6" containerID="3622a39dd6f9d77367e5fddf40cbc006bb584d664b04601b0cd315d7ac9c984c" exitCode=0
Sep 17 03:21:24.520736 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:24.520646    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c7659966b-292d6"
Sep 17 03:21:24.520736 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:24.520663    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c7659966b-292d6" event={"ID":"f88b24fe-fd7e-4c77-8648-423434670be6","Type":"ContainerDied","Data":"3622a39dd6f9d77367e5fddf40cbc006bb584d664b04601b0cd315d7ac9c984c"}
Sep 17 03:21:24.520736 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:24.520700    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c7659966b-292d6" event={"ID":"f88b24fe-fd7e-4c77-8648-423434670be6","Type":"ContainerDied","Data":"7b8aa995fbc65805a4315ecb1f5eccafb1489c559d387042320353fd2f7b9c50"}
Sep 17 03:21:24.520736 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:24.520721    2557 scope.go:117] "RemoveContainer" containerID="3622a39dd6f9d77367e5fddf40cbc006bb584d664b04601b0cd315d7ac9c984c"
Sep 17 03:21:24.529158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:24.529137    2557 scope.go:117] "RemoveContainer" containerID="3622a39dd6f9d77367e5fddf40cbc006bb584d664b04601b0cd315d7ac9c984c"
Sep 17 03:21:24.529454 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:21:24.529432    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3622a39dd6f9d77367e5fddf40cbc006bb584d664b04601b0cd315d7ac9c984c\": container with ID starting with 3622a39dd6f9d77367e5fddf40cbc006bb584d664b04601b0cd315d7ac9c984c not found: ID does not exist" containerID="3622a39dd6f9d77367e5fddf40cbc006bb584d664b04601b0cd315d7ac9c984c"
Sep 17 03:21:24.529526 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:24.529465    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3622a39dd6f9d77367e5fddf40cbc006bb584d664b04601b0cd315d7ac9c984c"} err="failed to get container status \"3622a39dd6f9d77367e5fddf40cbc006bb584d664b04601b0cd315d7ac9c984c\": rpc error: code = NotFound desc = could not find container \"3622a39dd6f9d77367e5fddf40cbc006bb584d664b04601b0cd315d7ac9c984c\": container with ID starting with 3622a39dd6f9d77367e5fddf40cbc006bb584d664b04601b0cd315d7ac9c984c not found: ID does not exist"
Sep 17 03:21:24.536911 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:24.536889    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c7659966b-292d6"]
Sep 17 03:21:24.540799 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:24.540779    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-c7659966b-292d6"]
Sep 17 03:21:25.913589 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:25.913556    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f88b24fe-fd7e-4c77-8648-423434670be6" path="/var/lib/kubelet/pods/f88b24fe-fd7e-4c77-8648-423434670be6/volumes"
Sep 17 03:21:31.918340 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:31.918309    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:21:31.922105 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:31.922078    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-9b7d74d76-fnw2t"
Sep 17 03:21:35.570677 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:35.570640    2557 generic.go:356] "Generic (PLEG): container finished" podID="8afb8ac0-6b66-48d0-afc1-32750519d672" containerID="3cfa0046a08875397000cfaee9478adc0c3917d628f1408c605c46b747fddaf1" exitCode=0
Sep 17 03:21:35.571076 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:35.570700    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg" event={"ID":"8afb8ac0-6b66-48d0-afc1-32750519d672","Type":"ContainerDied","Data":"3cfa0046a08875397000cfaee9478adc0c3917d628f1408c605c46b747fddaf1"}
Sep 17 03:21:35.571076 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:35.571005    2557 scope.go:117] "RemoveContainer" containerID="3cfa0046a08875397000cfaee9478adc0c3917d628f1408c605c46b747fddaf1"
Sep 17 03:21:36.575897 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:36.575863    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kmwgg" event={"ID":"8afb8ac0-6b66-48d0-afc1-32750519d672","Type":"ContainerStarted","Data":"ae55c3f05dbce8a430bbac997e212989fca49dd29e729be04049b3890beb78b2"}
Sep 17 03:21:37.532768 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:37.532737    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6685b49f78-sfnns_b1592c7c-2902-4b8d-9a15-b1bce854867f/router/0.log"
Sep 17 03:21:37.543318 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:37.543292    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gm8dh_100b3459-13e3-4558-96ad-dd122ec25b08/serve-healthcheck-canary/0.log"
Sep 17 03:21:53.810526 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:53.810484    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:21:53.827911 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:53.827881    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:21:54.668116 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:21:54.668085    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:11.888492 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:11.888446    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 03:22:11.889009 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:11.888770    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="kube-rbac-proxy" containerID="cri-o://1e4c6d7eba065950e39c632c996245f663589840c9873ad3411d5844961256d1" gracePeriod=600
Sep 17 03:22:11.889009 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:11.888791    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="config-reloader" containerID="cri-o://2c5d9612838d74a476ffd5ad8e8b47e401fad1064a560a7e4487dbf6af309d2a" gracePeriod=600
Sep 17 03:22:11.889009 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:11.888804    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="kube-rbac-proxy-web" containerID="cri-o://f9e2a0f0b02e60f8768db1eb9d40eccadeaffb66111987312db32c3f85f2826f" gracePeriod=600
Sep 17 03:22:11.889009 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:11.888807    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="thanos-sidecar" containerID="cri-o://9c383734cf9143552c30097adfd02784d86b16fd1b8847debec5bb6e00ad04f7" gracePeriod=600
Sep 17 03:22:11.889009 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:11.888897    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4f361545becd4a800f14e81e175e3e158b3e802572bdb37d28984d3953d50a94" gracePeriod=600
Sep 17 03:22:11.889009 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:11.888736    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="prometheus" containerID="cri-o://075c52ea3be9bd0d38cf9b69f4f43f8b2f32c33f7a16bb087c28479d3e2b3383" gracePeriod=600
Sep 17 03:22:12.726393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:12.726352    2557 generic.go:356] "Generic (PLEG): container finished" podID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerID="4f361545becd4a800f14e81e175e3e158b3e802572bdb37d28984d3953d50a94" exitCode=0
Sep 17 03:22:12.726393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:12.726378    2557 generic.go:356] "Generic (PLEG): container finished" podID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerID="1e4c6d7eba065950e39c632c996245f663589840c9873ad3411d5844961256d1" exitCode=0
Sep 17 03:22:12.726393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:12.726384    2557 generic.go:356] "Generic (PLEG): container finished" podID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerID="9c383734cf9143552c30097adfd02784d86b16fd1b8847debec5bb6e00ad04f7" exitCode=0
Sep 17 03:22:12.726393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:12.726390    2557 generic.go:356] "Generic (PLEG): container finished" podID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerID="2c5d9612838d74a476ffd5ad8e8b47e401fad1064a560a7e4487dbf6af309d2a" exitCode=0
Sep 17 03:22:12.726393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:12.726397    2557 generic.go:356] "Generic (PLEG): container finished" podID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerID="075c52ea3be9bd0d38cf9b69f4f43f8b2f32c33f7a16bb087c28479d3e2b3383" exitCode=0
Sep 17 03:22:12.726681 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:12.726419    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6beac4c9-90c9-42a2-8dac-15062d2a63f2","Type":"ContainerDied","Data":"4f361545becd4a800f14e81e175e3e158b3e802572bdb37d28984d3953d50a94"}
Sep 17 03:22:12.726681 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:12.726483    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6beac4c9-90c9-42a2-8dac-15062d2a63f2","Type":"ContainerDied","Data":"1e4c6d7eba065950e39c632c996245f663589840c9873ad3411d5844961256d1"}
Sep 17 03:22:12.726681 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:12.726511    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6beac4c9-90c9-42a2-8dac-15062d2a63f2","Type":"ContainerDied","Data":"9c383734cf9143552c30097adfd02784d86b16fd1b8847debec5bb6e00ad04f7"}
Sep 17 03:22:12.726681 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:12.726536    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6beac4c9-90c9-42a2-8dac-15062d2a63f2","Type":"ContainerDied","Data":"2c5d9612838d74a476ffd5ad8e8b47e401fad1064a560a7e4487dbf6af309d2a"}
Sep 17 03:22:12.726681 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:12.726561    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6beac4c9-90c9-42a2-8dac-15062d2a63f2","Type":"ContainerDied","Data":"075c52ea3be9bd0d38cf9b69f4f43f8b2f32c33f7a16bb087c28479d3e2b3383"}
Sep 17 03:22:13.143116 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.143095    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.223134 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223105    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-1\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223329 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223148    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-2\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223329 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223187    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-web-config\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223329 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223236    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-serving-certs-ca-bundle\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223329 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223266    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-thanos-prometheus-http-client-file\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223329 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223294    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6beac4c9-90c9-42a2-8dac-15062d2a63f2-config-out\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223329 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223327    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-tls\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223356    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223383    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-kubelet-serving-ca-bundle\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223420    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6beac4c9-90c9-42a2-8dac-15062d2a63f2-tls-assets\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223444    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-0\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223473    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223501    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs4b6\" (UniqueName: \"kubernetes.io/projected/6beac4c9-90c9-42a2-8dac-15062d2a63f2-kube-api-access-qs4b6\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223503    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:22:13.223656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223526    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-metrics-client-certs\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223554    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-metrics-client-ca\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223566    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:22:13.223656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223578    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-db\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223613    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-grpc-tls\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.223656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223637    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-kube-rbac-proxy\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.224301 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223668    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-config\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.224301 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223722    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-trusted-ca-bundle\") pod \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\" (UID: \"6beac4c9-90c9-42a2-8dac-15062d2a63f2\") "
Sep 17 03:22:13.224301 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223876    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:22:13.224301 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223914    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:22:13.224301 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223940    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.224301 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223958    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.224301 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.223973    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.224989 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.224961    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:22:13.225804 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.225397    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:22:13.226788 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.226257    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:22:13.227782 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.227659    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-config" (OuterVolumeSpecName: "config") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:13.228008 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.227986    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:22:13.228125 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.228007    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:13.228234 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.228171    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:13.228346 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.228167    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6beac4c9-90c9-42a2-8dac-15062d2a63f2-config-out" (OuterVolumeSpecName: "config-out") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:22:13.228448 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.228258    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6beac4c9-90c9-42a2-8dac-15062d2a63f2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:22:13.228961 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.228925    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:13.229220 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.229104    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6beac4c9-90c9-42a2-8dac-15062d2a63f2-kube-api-access-qs4b6" (OuterVolumeSpecName: "kube-api-access-qs4b6") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "kube-api-access-qs4b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:22:13.229220 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.229171    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:13.229220 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.229210    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:13.229775 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.229756    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:13.230087 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.230054    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:13.239619 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.239592    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-web-config" (OuterVolumeSpecName: "web-config") pod "6beac4c9-90c9-42a2-8dac-15062d2a63f2" (UID: "6beac4c9-90c9-42a2-8dac-15062d2a63f2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:22:13.324291 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324268    2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6beac4c9-90c9-42a2-8dac-15062d2a63f2-config-out\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324291 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324288    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324404 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324300    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324404 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324309    2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6beac4c9-90c9-42a2-8dac-15062d2a63f2-tls-assets\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324404 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324317    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324404 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324326    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324404 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324334    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs4b6\" (UniqueName: \"kubernetes.io/projected/6beac4c9-90c9-42a2-8dac-15062d2a63f2-kube-api-access-qs4b6\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324404 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324342    2557 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-metrics-client-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324404 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324351    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-metrics-client-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324404 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324360    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-k8s-db\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324404 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324368    2557 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-grpc-tls\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324404 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324376    2557 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-secret-kube-rbac-proxy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324404 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324384    2557 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-config\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324404 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324392    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324404 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324399    2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-web-config\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324404 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324408    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6beac4c9-90c9-42a2-8dac-15062d2a63f2-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.324905 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.324425    2557 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6beac4c9-90c9-42a2-8dac-15062d2a63f2-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:22:13.734455 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.734410    2557 generic.go:356] "Generic (PLEG): container finished" podID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerID="f9e2a0f0b02e60f8768db1eb9d40eccadeaffb66111987312db32c3f85f2826f" exitCode=0
Sep 17 03:22:13.734566 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.734503    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6beac4c9-90c9-42a2-8dac-15062d2a63f2","Type":"ContainerDied","Data":"f9e2a0f0b02e60f8768db1eb9d40eccadeaffb66111987312db32c3f85f2826f"}
Sep 17 03:22:13.734566 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.734535    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.734685 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.734615    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6beac4c9-90c9-42a2-8dac-15062d2a63f2","Type":"ContainerDied","Data":"a2b882225f21693b115fd8bd5df8b0721dd8072cfc49161524f0a43ff1e3a5d7"}
Sep 17 03:22:13.734685 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.734657    2557 scope.go:117] "RemoveContainer" containerID="4f361545becd4a800f14e81e175e3e158b3e802572bdb37d28984d3953d50a94"
Sep 17 03:22:13.743806 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.743785    2557 scope.go:117] "RemoveContainer" containerID="1e4c6d7eba065950e39c632c996245f663589840c9873ad3411d5844961256d1"
Sep 17 03:22:13.751154 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.751135    2557 scope.go:117] "RemoveContainer" containerID="f9e2a0f0b02e60f8768db1eb9d40eccadeaffb66111987312db32c3f85f2826f"
Sep 17 03:22:13.757846 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.757819    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 03:22:13.758505 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.758483    2557 scope.go:117] "RemoveContainer" containerID="9c383734cf9143552c30097adfd02784d86b16fd1b8847debec5bb6e00ad04f7"
Sep 17 03:22:13.762435 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.762411    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 03:22:13.766221 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.766183    2557 scope.go:117] "RemoveContainer" containerID="2c5d9612838d74a476ffd5ad8e8b47e401fad1064a560a7e4487dbf6af309d2a"
Sep 17 03:22:13.772941 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.772923    2557 scope.go:117] "RemoveContainer" containerID="075c52ea3be9bd0d38cf9b69f4f43f8b2f32c33f7a16bb087c28479d3e2b3383"
Sep 17 03:22:13.780241 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.780183    2557 scope.go:117] "RemoveContainer" containerID="09edf0d5e96db2aee6085a377759c42f06976cbfdcf8cf79239246a5ab4bd226"
Sep 17 03:22:13.787305 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.787288    2557 scope.go:117] "RemoveContainer" containerID="4f361545becd4a800f14e81e175e3e158b3e802572bdb37d28984d3953d50a94"
Sep 17 03:22:13.787600 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:22:13.787580    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f361545becd4a800f14e81e175e3e158b3e802572bdb37d28984d3953d50a94\": container with ID starting with 4f361545becd4a800f14e81e175e3e158b3e802572bdb37d28984d3953d50a94 not found: ID does not exist" containerID="4f361545becd4a800f14e81e175e3e158b3e802572bdb37d28984d3953d50a94"
Sep 17 03:22:13.787662 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.787610    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f361545becd4a800f14e81e175e3e158b3e802572bdb37d28984d3953d50a94"} err="failed to get container status \"4f361545becd4a800f14e81e175e3e158b3e802572bdb37d28984d3953d50a94\": rpc error: code = NotFound desc = could not find container \"4f361545becd4a800f14e81e175e3e158b3e802572bdb37d28984d3953d50a94\": container with ID starting with 4f361545becd4a800f14e81e175e3e158b3e802572bdb37d28984d3953d50a94 not found: ID does not exist"
Sep 17 03:22:13.787662 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.787627    2557 scope.go:117] "RemoveContainer" containerID="1e4c6d7eba065950e39c632c996245f663589840c9873ad3411d5844961256d1"
Sep 17 03:22:13.787872 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:22:13.787856    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e4c6d7eba065950e39c632c996245f663589840c9873ad3411d5844961256d1\": container with ID starting with 1e4c6d7eba065950e39c632c996245f663589840c9873ad3411d5844961256d1 not found: ID does not exist" containerID="1e4c6d7eba065950e39c632c996245f663589840c9873ad3411d5844961256d1"
Sep 17 03:22:13.787924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.787882    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e4c6d7eba065950e39c632c996245f663589840c9873ad3411d5844961256d1"} err="failed to get container status \"1e4c6d7eba065950e39c632c996245f663589840c9873ad3411d5844961256d1\": rpc error: code = NotFound desc = could not find container \"1e4c6d7eba065950e39c632c996245f663589840c9873ad3411d5844961256d1\": container with ID starting with 1e4c6d7eba065950e39c632c996245f663589840c9873ad3411d5844961256d1 not found: ID does not exist"
Sep 17 03:22:13.787924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.787897    2557 scope.go:117] "RemoveContainer" containerID="f9e2a0f0b02e60f8768db1eb9d40eccadeaffb66111987312db32c3f85f2826f"
Sep 17 03:22:13.787995 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.787967    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 03:22:13.788187 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:22:13.788168    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e2a0f0b02e60f8768db1eb9d40eccadeaffb66111987312db32c3f85f2826f\": container with ID starting with f9e2a0f0b02e60f8768db1eb9d40eccadeaffb66111987312db32c3f85f2826f not found: ID does not exist" containerID="f9e2a0f0b02e60f8768db1eb9d40eccadeaffb66111987312db32c3f85f2826f"
Sep 17 03:22:13.788258 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788227    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e2a0f0b02e60f8768db1eb9d40eccadeaffb66111987312db32c3f85f2826f"} err="failed to get container status \"f9e2a0f0b02e60f8768db1eb9d40eccadeaffb66111987312db32c3f85f2826f\": rpc error: code = NotFound desc = could not find container \"f9e2a0f0b02e60f8768db1eb9d40eccadeaffb66111987312db32c3f85f2826f\": container with ID starting with f9e2a0f0b02e60f8768db1eb9d40eccadeaffb66111987312db32c3f85f2826f not found: ID does not exist"
Sep 17 03:22:13.788258 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788250    2557 scope.go:117] "RemoveContainer" containerID="9c383734cf9143552c30097adfd02784d86b16fd1b8847debec5bb6e00ad04f7"
Sep 17 03:22:13.788332 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788308    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="kube-rbac-proxy-web"
Sep 17 03:22:13.788332 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788323    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="kube-rbac-proxy-web"
Sep 17 03:22:13.788423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788335    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="init-config-reloader"
Sep 17 03:22:13.788423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788341    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="init-config-reloader"
Sep 17 03:22:13.788423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788350    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="kube-rbac-proxy-thanos"
Sep 17 03:22:13.788423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788356    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="kube-rbac-proxy-thanos"
Sep 17 03:22:13.788423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788372    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="kube-rbac-proxy"
Sep 17 03:22:13.788423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788381    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="kube-rbac-proxy"
Sep 17 03:22:13.788423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788390    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="prometheus"
Sep 17 03:22:13.788423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788395    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="prometheus"
Sep 17 03:22:13.788423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788403    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="thanos-sidecar"
Sep 17 03:22:13.788423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788408    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="thanos-sidecar"
Sep 17 03:22:13.788423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788425    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="config-reloader"
Sep 17 03:22:13.788845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788434    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="config-reloader"
Sep 17 03:22:13.788845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788445    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f88b24fe-fd7e-4c77-8648-423434670be6" containerName="registry"
Sep 17 03:22:13.788845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788454    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f88b24fe-fd7e-4c77-8648-423434670be6" containerName="registry"
Sep 17 03:22:13.788845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788513    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="prometheus"
Sep 17 03:22:13.788845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788523    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f88b24fe-fd7e-4c77-8648-423434670be6" containerName="registry"
Sep 17 03:22:13.788845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788534    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="kube-rbac-proxy"
Sep 17 03:22:13.788845 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:22:13.788537    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c383734cf9143552c30097adfd02784d86b16fd1b8847debec5bb6e00ad04f7\": container with ID starting with 9c383734cf9143552c30097adfd02784d86b16fd1b8847debec5bb6e00ad04f7 not found: ID does not exist" containerID="9c383734cf9143552c30097adfd02784d86b16fd1b8847debec5bb6e00ad04f7"
Sep 17 03:22:13.788845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788559    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c383734cf9143552c30097adfd02784d86b16fd1b8847debec5bb6e00ad04f7"} err="failed to get container status \"9c383734cf9143552c30097adfd02784d86b16fd1b8847debec5bb6e00ad04f7\": rpc error: code = NotFound desc = could not find container \"9c383734cf9143552c30097adfd02784d86b16fd1b8847debec5bb6e00ad04f7\": container with ID starting with 9c383734cf9143552c30097adfd02784d86b16fd1b8847debec5bb6e00ad04f7 not found: ID does not exist"
Sep 17 03:22:13.788845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788572    2557 scope.go:117] "RemoveContainer" containerID="2c5d9612838d74a476ffd5ad8e8b47e401fad1064a560a7e4487dbf6af309d2a"
Sep 17 03:22:13.788845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788543    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="kube-rbac-proxy-thanos"
Sep 17 03:22:13.788845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788623    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="config-reloader"
Sep 17 03:22:13.788845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788635    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="thanos-sidecar"
Sep 17 03:22:13.788845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788645    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" containerName="kube-rbac-proxy-web"
Sep 17 03:22:13.789373 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:22:13.788859    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5d9612838d74a476ffd5ad8e8b47e401fad1064a560a7e4487dbf6af309d2a\": container with ID starting with 2c5d9612838d74a476ffd5ad8e8b47e401fad1064a560a7e4487dbf6af309d2a not found: ID does not exist" containerID="2c5d9612838d74a476ffd5ad8e8b47e401fad1064a560a7e4487dbf6af309d2a"
Sep 17 03:22:13.789373 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788889    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5d9612838d74a476ffd5ad8e8b47e401fad1064a560a7e4487dbf6af309d2a"} err="failed to get container status \"2c5d9612838d74a476ffd5ad8e8b47e401fad1064a560a7e4487dbf6af309d2a\": rpc error: code = NotFound desc = could not find container \"2c5d9612838d74a476ffd5ad8e8b47e401fad1064a560a7e4487dbf6af309d2a\": container with ID starting with 2c5d9612838d74a476ffd5ad8e8b47e401fad1064a560a7e4487dbf6af309d2a not found: ID does not exist"
Sep 17 03:22:13.789373 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.788910    2557 scope.go:117] "RemoveContainer" containerID="075c52ea3be9bd0d38cf9b69f4f43f8b2f32c33f7a16bb087c28479d3e2b3383"
Sep 17 03:22:13.789373 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:22:13.789147    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"075c52ea3be9bd0d38cf9b69f4f43f8b2f32c33f7a16bb087c28479d3e2b3383\": container with ID starting with 075c52ea3be9bd0d38cf9b69f4f43f8b2f32c33f7a16bb087c28479d3e2b3383 not found: ID does not exist" containerID="075c52ea3be9bd0d38cf9b69f4f43f8b2f32c33f7a16bb087c28479d3e2b3383"
Sep 17 03:22:13.789373 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.789165    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"075c52ea3be9bd0d38cf9b69f4f43f8b2f32c33f7a16bb087c28479d3e2b3383"} err="failed to get container status \"075c52ea3be9bd0d38cf9b69f4f43f8b2f32c33f7a16bb087c28479d3e2b3383\": rpc error: code = NotFound desc = could not find container \"075c52ea3be9bd0d38cf9b69f4f43f8b2f32c33f7a16bb087c28479d3e2b3383\": container with ID starting with 075c52ea3be9bd0d38cf9b69f4f43f8b2f32c33f7a16bb087c28479d3e2b3383 not found: ID does not exist"
Sep 17 03:22:13.789373 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.789180    2557 scope.go:117] "RemoveContainer" containerID="09edf0d5e96db2aee6085a377759c42f06976cbfdcf8cf79239246a5ab4bd226"
Sep 17 03:22:13.789605 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:22:13.789563    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09edf0d5e96db2aee6085a377759c42f06976cbfdcf8cf79239246a5ab4bd226\": container with ID starting with 09edf0d5e96db2aee6085a377759c42f06976cbfdcf8cf79239246a5ab4bd226 not found: ID does not exist" containerID="09edf0d5e96db2aee6085a377759c42f06976cbfdcf8cf79239246a5ab4bd226"
Sep 17 03:22:13.789605 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.789590    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09edf0d5e96db2aee6085a377759c42f06976cbfdcf8cf79239246a5ab4bd226"} err="failed to get container status \"09edf0d5e96db2aee6085a377759c42f06976cbfdcf8cf79239246a5ab4bd226\": rpc error: code = NotFound desc = could not find container \"09edf0d5e96db2aee6085a377759c42f06976cbfdcf8cf79239246a5ab4bd226\": container with ID starting with 09edf0d5e96db2aee6085a377759c42f06976cbfdcf8cf79239246a5ab4bd226 not found: ID does not exist"
Sep 17 03:22:13.793767 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.793752    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.795911 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.795892    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 03:22:13.795999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.795908    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 03:22:13.796283 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.796252    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 03:22:13.796360 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.796256    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r2jjr\""
Sep 17 03:22:13.796360 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.796296    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 03:22:13.796639 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.796608    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 03:22:13.796740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.796701    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-evli7s2s64m73\""
Sep 17 03:22:13.796740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.796730    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 03:22:13.796931 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.796742    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 03:22:13.796931 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.796874    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 03:22:13.796931 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.796905    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 03:22:13.797076 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.796982    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 03:22:13.797150 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.797129    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 03:22:13.797383 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.797361    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 03:22:13.799502 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.799478    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 03:22:13.802186 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.802162    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 03:22:13.806028 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.806009    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 03:22:13.913122 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.913095    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6beac4c9-90c9-42a2-8dac-15062d2a63f2" path="/var/lib/kubelet/pods/6beac4c9-90c9-42a2-8dac-15062d2a63f2/volumes"
Sep 17 03:22:13.928497 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928476    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-config-out\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928576 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928501    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928576 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928523    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928576 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928539    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-config\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928576 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928563    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-web-config\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928721 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928587    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928721 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928608    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928721 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928624    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928721 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928643    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928721 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928665    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6slc\" (UniqueName: \"kubernetes.io/projected/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-kube-api-access-h6slc\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928721 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928681    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928721 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928697    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928721 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928717    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928974 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928768    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928974 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928817    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928974 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928836    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928974 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928853    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928974 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928875    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928974 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928910    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:13.928974 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:13.928935    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030151 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030100    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030151 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030128    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030151 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030149    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030376 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030167    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030376 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030186    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030376 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030233    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030376 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030256    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-config-out\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030376 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030273    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030376 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030309    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030376 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030336    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-config\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030376 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030367    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-web-config\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030397    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030428    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030452    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030476    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030507    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6slc\" (UniqueName: \"kubernetes.io/projected/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-kube-api-access-h6slc\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030548    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030576    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030611    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.030824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030642    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.031290 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030858    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.031290 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.030909    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.031776 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.031625    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.031956 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.031932    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.031956 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.031958    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.032311 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.032286    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.032682 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.032657    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.033592 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.033565    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.033592 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.033572    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.033838 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.033830    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.034661 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.034637    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.034814 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.034759    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.035170 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.035021    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.035170 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.035087    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-config-out\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.035486 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.035468    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-web-config\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.035682 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.035657    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-config\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.036886 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.036867    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.037062 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.037040    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.037131 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.037063    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.040111 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.040089    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6slc\" (UniqueName: \"kubernetes.io/projected/cabbe6ab-13bc-4a45-938e-fdd788d9afbd-kube-api-access-h6slc\") pod \"prometheus-k8s-0\" (UID: \"cabbe6ab-13bc-4a45-938e-fdd788d9afbd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.104361 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.104339    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:22:14.385017 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.384995    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 03:22:14.386406 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:22:14.386375    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabbe6ab_13bc_4a45_938e_fdd788d9afbd.slice/crio-713ebf6d4d92d61b6367caf0454156df4d3e1ffcbaf7654b6b9e2f439a9fcf7b WatchSource:0}: Error finding container 713ebf6d4d92d61b6367caf0454156df4d3e1ffcbaf7654b6b9e2f439a9fcf7b: Status 404 returned error can't find the container with id 713ebf6d4d92d61b6367caf0454156df4d3e1ffcbaf7654b6b9e2f439a9fcf7b
Sep 17 03:22:14.741345 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.741270    2557 generic.go:356] "Generic (PLEG): container finished" podID="cabbe6ab-13bc-4a45-938e-fdd788d9afbd" containerID="f05bf146e7b05b830f058e1587385b63dc1175964b3912e996d411db771d1d4e" exitCode=0
Sep 17 03:22:14.741438 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.741343    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cabbe6ab-13bc-4a45-938e-fdd788d9afbd","Type":"ContainerDied","Data":"f05bf146e7b05b830f058e1587385b63dc1175964b3912e996d411db771d1d4e"}
Sep 17 03:22:14.741438 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:14.741404    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cabbe6ab-13bc-4a45-938e-fdd788d9afbd","Type":"ContainerStarted","Data":"713ebf6d4d92d61b6367caf0454156df4d3e1ffcbaf7654b6b9e2f439a9fcf7b"}
Sep 17 03:22:15.749130 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:15.749091    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cabbe6ab-13bc-4a45-938e-fdd788d9afbd","Type":"ContainerStarted","Data":"0161aa2ec0eb6fcdbf82139b1762ed9440f7a7da3dc9fd8ec1df4bd26558d489"}
Sep 17 03:22:15.749526 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:15.749161    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cabbe6ab-13bc-4a45-938e-fdd788d9afbd","Type":"ContainerStarted","Data":"32e327e6ab02aee1b9addc44a63761fbd77240a834cece29191f832834c52fd9"}
Sep 17 03:22:15.749526 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:15.749208    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cabbe6ab-13bc-4a45-938e-fdd788d9afbd","Type":"ContainerStarted","Data":"72c3d326169bc9bf152eb7d11166f4ca7576ab5328678defab23ff60872c4ed4"}
Sep 17 03:22:15.749526 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:15.749237    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cabbe6ab-13bc-4a45-938e-fdd788d9afbd","Type":"ContainerStarted","Data":"c379395dead8330a01a1b619f2956e505e809e5df1948a7408b3c7246156d64c"}
Sep 17 03:22:15.749526 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:15.749260    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cabbe6ab-13bc-4a45-938e-fdd788d9afbd","Type":"ContainerStarted","Data":"ca3d878bcfffc5a403936dcdcbb5ce96285e2d0bfb3b7425df74572a89b2fdd8"}
Sep 17 03:22:15.749526 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:15.749298    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cabbe6ab-13bc-4a45-938e-fdd788d9afbd","Type":"ContainerStarted","Data":"70b9f8cf6667cff59fe430226e25c1337689e41d2cea4e5c60a0f35f77b3b3f0"}
Sep 17 03:22:15.777745 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:15.777694    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7528137839999998 podStartE2EDuration="2.777678202s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.864418ms" isStatefulPod=false podCreationTimestamp="2026-09-17 03:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:22:15.77464809 +0000 UTC m=+186.377744120" watchObservedRunningTime="2026-09-17 03:22:15.777678202 +0000 UTC m=+186.380774203"
Sep 17 03:22:19.105168 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:22:19.105130    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:23:14.105291 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:14.105258    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:23:14.120062 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:14.120041    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:23:15.010866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:15.010842    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 03:23:55.071131 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:55.071092    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fvd82"]
Sep 17 03:23:55.074551 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:55.074529    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fvd82"
Sep 17 03:23:55.076499 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:55.076476    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 03:23:55.076612 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:55.076480    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 03:23:55.077097 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:55.077079    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-c96wc\""
Sep 17 03:23:55.083117 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:55.083096    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fvd82"]
Sep 17 03:23:55.143895 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:55.143868    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtn2l\" (UniqueName: \"kubernetes.io/projected/36af1d69-9fe1-44b8-b02e-c609ed54a9e2-kube-api-access-dtn2l\") pod \"cert-manager-55b4d954d5-fvd82\" (UID: \"36af1d69-9fe1-44b8-b02e-c609ed54a9e2\") " pod="cert-manager/cert-manager-55b4d954d5-fvd82"
Sep 17 03:23:55.143895 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:55.143891    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36af1d69-9fe1-44b8-b02e-c609ed54a9e2-bound-sa-token\") pod \"cert-manager-55b4d954d5-fvd82\" (UID: \"36af1d69-9fe1-44b8-b02e-c609ed54a9e2\") " pod="cert-manager/cert-manager-55b4d954d5-fvd82"
Sep 17 03:23:55.245138 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:55.245113    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtn2l\" (UniqueName: \"kubernetes.io/projected/36af1d69-9fe1-44b8-b02e-c609ed54a9e2-kube-api-access-dtn2l\") pod \"cert-manager-55b4d954d5-fvd82\" (UID: \"36af1d69-9fe1-44b8-b02e-c609ed54a9e2\") " pod="cert-manager/cert-manager-55b4d954d5-fvd82"
Sep 17 03:23:55.245252 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:55.245143    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36af1d69-9fe1-44b8-b02e-c609ed54a9e2-bound-sa-token\") pod \"cert-manager-55b4d954d5-fvd82\" (UID: \"36af1d69-9fe1-44b8-b02e-c609ed54a9e2\") " pod="cert-manager/cert-manager-55b4d954d5-fvd82"
Sep 17 03:23:55.252377 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:55.252357    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtn2l\" (UniqueName: \"kubernetes.io/projected/36af1d69-9fe1-44b8-b02e-c609ed54a9e2-kube-api-access-dtn2l\") pod \"cert-manager-55b4d954d5-fvd82\" (UID: \"36af1d69-9fe1-44b8-b02e-c609ed54a9e2\") " pod="cert-manager/cert-manager-55b4d954d5-fvd82"
Sep 17 03:23:55.252463 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:55.252363    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36af1d69-9fe1-44b8-b02e-c609ed54a9e2-bound-sa-token\") pod \"cert-manager-55b4d954d5-fvd82\" (UID: \"36af1d69-9fe1-44b8-b02e-c609ed54a9e2\") " pod="cert-manager/cert-manager-55b4d954d5-fvd82"
Sep 17 03:23:55.395783 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:55.395762    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fvd82"
Sep 17 03:23:55.660847 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:55.660820    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fvd82"]
Sep 17 03:23:55.663422 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:23:55.663387    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36af1d69_9fe1_44b8_b02e_c609ed54a9e2.slice/crio-1e86071933a105ee3e59702b62c84b35a94a06d8fee84856664cb52be236bdfb WatchSource:0}: Error finding container 1e86071933a105ee3e59702b62c84b35a94a06d8fee84856664cb52be236bdfb: Status 404 returned error can't find the container with id 1e86071933a105ee3e59702b62c84b35a94a06d8fee84856664cb52be236bdfb
Sep 17 03:23:56.161180 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:56.161148    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fvd82" event={"ID":"36af1d69-9fe1-44b8-b02e-c609ed54a9e2","Type":"ContainerStarted","Data":"1e86071933a105ee3e59702b62c84b35a94a06d8fee84856664cb52be236bdfb"}
Sep 17 03:23:59.176888 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:59.176854    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fvd82" event={"ID":"36af1d69-9fe1-44b8-b02e-c609ed54a9e2","Type":"ContainerStarted","Data":"e1dd3384b277e6d65cd2afeb598e100c70cf5a8e272211cd44451cd07546205a"}
Sep 17 03:23:59.193985 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:23:59.193941    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-fvd82" podStartSLOduration=1.138723609 podStartE2EDuration="4.193926377s" totalImagesPullingTime="3.055202768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:23:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:23:59.192625896 +0000 UTC m=+289.795721901" watchObservedRunningTime="2026-09-17 03:23:59.193926377 +0000 UTC m=+289.797022381"
Sep 17 03:24:09.818400 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:09.818371    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:24:09.823374 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:09.823348    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:24:09.831100 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:09.831075    2557 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 03:24:19.767440 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:19.767399    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"]
Sep 17 03:24:19.771889 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:19.771868    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:19.774028 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:19.774003    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 03:24:19.774132 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:19.774006    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8vv95\""
Sep 17 03:24:19.774205 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:19.774133    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 03:24:19.774451 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:19.774427    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 03:24:19.774547 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:19.774529    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 03:24:19.791055 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:19.791031    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"]
Sep 17 03:24:19.916341 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:19.916317    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89bd3a1e-642e-4389-a7b4-e0407a99cd8e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-f8wcx\" (UID: \"89bd3a1e-642e-4389-a7b4-e0407a99cd8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:19.916459 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:19.916352    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbrgd\" (UniqueName: \"kubernetes.io/projected/89bd3a1e-642e-4389-a7b4-e0407a99cd8e-kube-api-access-vbrgd\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-f8wcx\" (UID: \"89bd3a1e-642e-4389-a7b4-e0407a99cd8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:19.916459 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:19.916372    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/89bd3a1e-642e-4389-a7b4-e0407a99cd8e-manifests\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-f8wcx\" (UID: \"89bd3a1e-642e-4389-a7b4-e0407a99cd8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:19.916547 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:19.916462    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/89bd3a1e-642e-4389-a7b4-e0407a99cd8e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-f8wcx\" (UID: \"89bd3a1e-642e-4389-a7b4-e0407a99cd8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:19.916547 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:19.916501    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89bd3a1e-642e-4389-a7b4-e0407a99cd8e-tmp\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-f8wcx\" (UID: \"89bd3a1e-642e-4389-a7b4-e0407a99cd8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:20.017401 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:20.017378    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89bd3a1e-642e-4389-a7b4-e0407a99cd8e-tmp\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-f8wcx\" (UID: \"89bd3a1e-642e-4389-a7b4-e0407a99cd8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:20.017520 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:20.017415    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89bd3a1e-642e-4389-a7b4-e0407a99cd8e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-f8wcx\" (UID: \"89bd3a1e-642e-4389-a7b4-e0407a99cd8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:20.017520 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:20.017440    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbrgd\" (UniqueName: \"kubernetes.io/projected/89bd3a1e-642e-4389-a7b4-e0407a99cd8e-kube-api-access-vbrgd\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-f8wcx\" (UID: \"89bd3a1e-642e-4389-a7b4-e0407a99cd8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:20.017520 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:20.017468    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/89bd3a1e-642e-4389-a7b4-e0407a99cd8e-manifests\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-f8wcx\" (UID: \"89bd3a1e-642e-4389-a7b4-e0407a99cd8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:20.017520 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:20.017517    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/89bd3a1e-642e-4389-a7b4-e0407a99cd8e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-f8wcx\" (UID: \"89bd3a1e-642e-4389-a7b4-e0407a99cd8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:20.017853 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:20.017831    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/89bd3a1e-642e-4389-a7b4-e0407a99cd8e-manifests\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-f8wcx\" (UID: \"89bd3a1e-642e-4389-a7b4-e0407a99cd8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:20.019425 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:20.019400    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89bd3a1e-642e-4389-a7b4-e0407a99cd8e-tmp\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-f8wcx\" (UID: \"89bd3a1e-642e-4389-a7b4-e0407a99cd8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:20.019835 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:20.019819    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/89bd3a1e-642e-4389-a7b4-e0407a99cd8e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-f8wcx\" (UID: \"89bd3a1e-642e-4389-a7b4-e0407a99cd8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:20.019900 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:20.019879    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89bd3a1e-642e-4389-a7b4-e0407a99cd8e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-f8wcx\" (UID: \"89bd3a1e-642e-4389-a7b4-e0407a99cd8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:20.024182 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:20.024162    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbrgd\" (UniqueName: \"kubernetes.io/projected/89bd3a1e-642e-4389-a7b4-e0407a99cd8e-kube-api-access-vbrgd\") pod \"opendatahub-operator-controller-manager-6977f7b7bc-f8wcx\" (UID: \"89bd3a1e-642e-4389-a7b4-e0407a99cd8e\") " pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:20.083893 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:20.083872    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:20.372277 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:20.372252    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"]
Sep 17 03:24:20.374563 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:24:20.374540    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bd3a1e_642e_4389_a7b4_e0407a99cd8e.slice/crio-af09ab54a6d6bedb0e0831d459486e5f61eb7f2d94e0e63e438ba3017e9b7ed1 WatchSource:0}: Error finding container af09ab54a6d6bedb0e0831d459486e5f61eb7f2d94e0e63e438ba3017e9b7ed1: Status 404 returned error can't find the container with id af09ab54a6d6bedb0e0831d459486e5f61eb7f2d94e0e63e438ba3017e9b7ed1
Sep 17 03:24:20.377124 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:20.377103    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 03:24:21.265598 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:21.265554    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx" event={"ID":"89bd3a1e-642e-4389-a7b4-e0407a99cd8e","Type":"ContainerStarted","Data":"af09ab54a6d6bedb0e0831d459486e5f61eb7f2d94e0e63e438ba3017e9b7ed1"}
Sep 17 03:24:24.279502 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:24.279457    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx" event={"ID":"89bd3a1e-642e-4389-a7b4-e0407a99cd8e","Type":"ContainerStarted","Data":"d081a969c11cd9ac669b281cc617362fa9f84ab1f7137d09fed04bc6e0c7fa36"}
Sep 17 03:24:25.286208 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:25.286162    2557 generic.go:356] "Generic (PLEG): container finished" podID="89bd3a1e-642e-4389-a7b4-e0407a99cd8e" containerID="d081a969c11cd9ac669b281cc617362fa9f84ab1f7137d09fed04bc6e0c7fa36" exitCode=0
Sep 17 03:24:25.286583 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:25.286224    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx" event={"ID":"89bd3a1e-642e-4389-a7b4-e0407a99cd8e","Type":"ContainerDied","Data":"d081a969c11cd9ac669b281cc617362fa9f84ab1f7137d09fed04bc6e0c7fa36"}
Sep 17 03:24:26.292242 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:26.292189    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx" event={"ID":"89bd3a1e-642e-4389-a7b4-e0407a99cd8e","Type":"ContainerStarted","Data":"21df22e8ed860b2972acfa432f314a0b008eb2e6951b2b64ab85e32504d2d6e1"}
Sep 17 03:24:26.292612 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:26.292345    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:26.313224 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:26.313162    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx" podStartSLOduration=2.816889034 podStartE2EDuration="7.313146373s" totalImagesPullingTime="3.475490222s" totalInitContainerRuntime="1.020767117s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:24:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:24:26.312922955 +0000 UTC m=+316.916018960" watchObservedRunningTime="2026-09-17 03:24:26.313146373 +0000 UTC m=+316.916242376"
Sep 17 03:24:37.299764 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:37.299735    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6977f7b7bc-f8wcx"
Sep 17 03:24:45.279923 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.279889    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"]
Sep 17 03:24:45.283178 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.283161    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.285341 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.285317    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 03:24:45.285341 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.285336    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-wkrqw\""
Sep 17 03:24:45.293435 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.293406    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"]
Sep 17 03:24:45.303149 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.303127    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1066f870-0e47-474b-a5bc-68795aa2f3a3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.303263 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.303168    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1066f870-0e47-474b-a5bc-68795aa2f3a3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.303263 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.303189    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1066f870-0e47-474b-a5bc-68795aa2f3a3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.303347 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.303265    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1066f870-0e47-474b-a5bc-68795aa2f3a3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.303347 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.303285    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1066f870-0e47-474b-a5bc-68795aa2f3a3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.303347 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.303329    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1066f870-0e47-474b-a5bc-68795aa2f3a3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.303347 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.303346    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1066f870-0e47-474b-a5bc-68795aa2f3a3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.303498 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.303369    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5rbh\" (UniqueName: \"kubernetes.io/projected/1066f870-0e47-474b-a5bc-68795aa2f3a3-kube-api-access-h5rbh\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.303498 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.303400    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1066f870-0e47-474b-a5bc-68795aa2f3a3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.403712 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.403689    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1066f870-0e47-474b-a5bc-68795aa2f3a3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.403807 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.403727    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1066f870-0e47-474b-a5bc-68795aa2f3a3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.403807 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.403747    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1066f870-0e47-474b-a5bc-68795aa2f3a3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.403807 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.403783    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1066f870-0e47-474b-a5bc-68795aa2f3a3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.403913 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.403822    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1066f870-0e47-474b-a5bc-68795aa2f3a3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.403913 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.403863    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1066f870-0e47-474b-a5bc-68795aa2f3a3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.403913 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.403887    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1066f870-0e47-474b-a5bc-68795aa2f3a3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.404056 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.403914    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rbh\" (UniqueName: \"kubernetes.io/projected/1066f870-0e47-474b-a5bc-68795aa2f3a3-kube-api-access-h5rbh\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.404056 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.403952    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1066f870-0e47-474b-a5bc-68795aa2f3a3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.404332 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.404283    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1066f870-0e47-474b-a5bc-68795aa2f3a3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.404442 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.404330    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1066f870-0e47-474b-a5bc-68795aa2f3a3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.404512 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.404491    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1066f870-0e47-474b-a5bc-68795aa2f3a3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.404615 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.404597    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1066f870-0e47-474b-a5bc-68795aa2f3a3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.404698 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.404678    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1066f870-0e47-474b-a5bc-68795aa2f3a3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.406046 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.406025    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1066f870-0e47-474b-a5bc-68795aa2f3a3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.406229 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.406189    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1066f870-0e47-474b-a5bc-68795aa2f3a3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.415038 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.415012    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1066f870-0e47-474b-a5bc-68795aa2f3a3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.415314 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.415293    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5rbh\" (UniqueName: \"kubernetes.io/projected/1066f870-0e47-474b-a5bc-68795aa2f3a3-kube-api-access-h5rbh\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww\" (UID: \"1066f870-0e47-474b-a5bc-68795aa2f3a3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.594518 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.594456    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:45.867267 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:45.867243    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"]
Sep 17 03:24:45.869498 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:24:45.869472    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1066f870_0e47_474b_a5bc_68795aa2f3a3.slice/crio-5084a80c04ba3eb703bb275322b1d1e99f8e8279fbe7541a5faf2bd0914a0171 WatchSource:0}: Error finding container 5084a80c04ba3eb703bb275322b1d1e99f8e8279fbe7541a5faf2bd0914a0171: Status 404 returned error can't find the container with id 5084a80c04ba3eb703bb275322b1d1e99f8e8279fbe7541a5faf2bd0914a0171
Sep 17 03:24:46.383289 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:46.383261    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww" event={"ID":"1066f870-0e47-474b-a5bc-68795aa2f3a3","Type":"ContainerStarted","Data":"5084a80c04ba3eb703bb275322b1d1e99f8e8279fbe7541a5faf2bd0914a0171"}
Sep 17 03:24:48.466779 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:48.466739    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:24:48.467058 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:48.466813    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:24:48.467058 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:48.466844    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:24:49.401377 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:49.401331    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww" event={"ID":"1066f870-0e47-474b-a5bc-68795aa2f3a3","Type":"ContainerStarted","Data":"8c59de0671f7f932cdf5c73038b540354452317e98b33a8acf1759f24d123065"}
Sep 17 03:24:49.419250 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:49.419164    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww" podStartSLOduration=1.824148746 podStartE2EDuration="4.419148131s" totalImagesPullingTime="2.594999385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:24:49.417226265 +0000 UTC m=+340.020322268" watchObservedRunningTime="2026-09-17 03:24:49.419148131 +0000 UTC m=+340.022244134"
Sep 17 03:24:49.594945 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:49.594919    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:49.599240 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:49.599215    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:50.407085 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:50.407052    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:50.408292 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:50.408262    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww"
Sep 17 03:24:57.389490 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.389455    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"]
Sep 17 03:24:57.468014 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.467986    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"]
Sep 17 03:24:57.468158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.468071    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:24:57.471330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.471304    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 03:24:57.471475 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.471347    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 03:24:57.471475 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.471304    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 03:24:57.471475 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.471421    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gxnb5\""
Sep 17 03:24:57.471475 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.471464    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 03:24:57.471670 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.471564    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 03:24:57.486503 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.486482    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b33ba81b-b815-434c-a131-6e57c834c613-manager-config\") pod \"lws-controller-manager-55d557bdbb-lfwgd\" (UID: \"b33ba81b-b815-434c-a131-6e57c834c613\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:24:57.486604 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.486514    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b33ba81b-b815-434c-a131-6e57c834c613-cert\") pod \"lws-controller-manager-55d557bdbb-lfwgd\" (UID: \"b33ba81b-b815-434c-a131-6e57c834c613\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:24:57.486604 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.486547    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b33ba81b-b815-434c-a131-6e57c834c613-metrics-cert\") pod \"lws-controller-manager-55d557bdbb-lfwgd\" (UID: \"b33ba81b-b815-434c-a131-6e57c834c613\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:24:57.486679 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.486652    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hx96\" (UniqueName: \"kubernetes.io/projected/b33ba81b-b815-434c-a131-6e57c834c613-kube-api-access-8hx96\") pod \"lws-controller-manager-55d557bdbb-lfwgd\" (UID: \"b33ba81b-b815-434c-a131-6e57c834c613\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:24:57.587950 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.587925    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b33ba81b-b815-434c-a131-6e57c834c613-manager-config\") pod \"lws-controller-manager-55d557bdbb-lfwgd\" (UID: \"b33ba81b-b815-434c-a131-6e57c834c613\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:24:57.588166 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.588150    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b33ba81b-b815-434c-a131-6e57c834c613-cert\") pod \"lws-controller-manager-55d557bdbb-lfwgd\" (UID: \"b33ba81b-b815-434c-a131-6e57c834c613\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:24:57.588341 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.588326    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b33ba81b-b815-434c-a131-6e57c834c613-metrics-cert\") pod \"lws-controller-manager-55d557bdbb-lfwgd\" (UID: \"b33ba81b-b815-434c-a131-6e57c834c613\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:24:57.588500 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.588486    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hx96\" (UniqueName: \"kubernetes.io/projected/b33ba81b-b815-434c-a131-6e57c834c613-kube-api-access-8hx96\") pod \"lws-controller-manager-55d557bdbb-lfwgd\" (UID: \"b33ba81b-b815-434c-a131-6e57c834c613\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:24:57.589105 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.588750    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b33ba81b-b815-434c-a131-6e57c834c613-manager-config\") pod \"lws-controller-manager-55d557bdbb-lfwgd\" (UID: \"b33ba81b-b815-434c-a131-6e57c834c613\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:24:57.591653 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.591632    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b33ba81b-b815-434c-a131-6e57c834c613-metrics-cert\") pod \"lws-controller-manager-55d557bdbb-lfwgd\" (UID: \"b33ba81b-b815-434c-a131-6e57c834c613\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:24:57.592248 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.592179    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b33ba81b-b815-434c-a131-6e57c834c613-cert\") pod \"lws-controller-manager-55d557bdbb-lfwgd\" (UID: \"b33ba81b-b815-434c-a131-6e57c834c613\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:24:57.598067 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.598040    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hx96\" (UniqueName: \"kubernetes.io/projected/b33ba81b-b815-434c-a131-6e57c834c613-kube-api-access-8hx96\") pod \"lws-controller-manager-55d557bdbb-lfwgd\" (UID: \"b33ba81b-b815-434c-a131-6e57c834c613\") " pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:24:57.777364 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:57.777309    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:24:58.057839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.057806    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"]
Sep 17 03:24:58.058961 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:24:58.058929    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33ba81b_b815_434c_a131_6e57c834c613.slice/crio-da98f05a63c8fb1295431c81bbcffa582b759ce1a6b0c04792a95b8d4156cae6 WatchSource:0}: Error finding container da98f05a63c8fb1295431c81bbcffa582b759ce1a6b0c04792a95b8d4156cae6: Status 404 returned error can't find the container with id da98f05a63c8fb1295431c81bbcffa582b759ce1a6b0c04792a95b8d4156cae6
Sep 17 03:24:58.441987 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.441952    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd" event={"ID":"b33ba81b-b815-434c-a131-6e57c834c613","Type":"ContainerStarted","Data":"da98f05a63c8fb1295431c81bbcffa582b759ce1a6b0c04792a95b8d4156cae6"}
Sep 17 03:24:58.817310 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.817234    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-grtpb"]
Sep 17 03:24:58.820758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.820733    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-grtpb"
Sep 17 03:24:58.824173 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.824151    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vl9lj\""
Sep 17 03:24:58.824351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.824313    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 17 03:24:58.833555 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.833491    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-grtpb"]
Sep 17 03:24:58.837238 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.835268    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt"]
Sep 17 03:24:58.841495 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.840754    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt"
Sep 17 03:24:58.843086 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.842709    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-rjgxw\""
Sep 17 03:24:58.843086 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.842928    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 17 03:24:58.850945 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.850923    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt"]
Sep 17 03:24:58.900375 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.900349    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7-cert\") pod \"kserve-controller-manager-d6f4b65b8-grtpb\" (UID: \"6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-grtpb"
Sep 17 03:24:58.900561 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.900393    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvhkl\" (UniqueName: \"kubernetes.io/projected/0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8-kube-api-access-xvhkl\") pod \"llmisvc-controller-manager-777c6bd7d5-tcntt\" (UID: \"0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt"
Sep 17 03:24:58.900561 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.900451    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx6dr\" (UniqueName: \"kubernetes.io/projected/6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7-kube-api-access-lx6dr\") pod \"kserve-controller-manager-d6f4b65b8-grtpb\" (UID: \"6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-grtpb"
Sep 17 03:24:58.900561 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:58.900503    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tcntt\" (UID: \"0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt"
Sep 17 03:24:59.004687 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:59.003577    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tcntt\" (UID: \"0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt"
Sep 17 03:24:59.004687 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:59.003669    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7-cert\") pod \"kserve-controller-manager-d6f4b65b8-grtpb\" (UID: \"6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-grtpb"
Sep 17 03:24:59.004687 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:59.003698    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvhkl\" (UniqueName: \"kubernetes.io/projected/0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8-kube-api-access-xvhkl\") pod \"llmisvc-controller-manager-777c6bd7d5-tcntt\" (UID: \"0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt"
Sep 17 03:24:59.004687 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:59.003750    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx6dr\" (UniqueName: \"kubernetes.io/projected/6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7-kube-api-access-lx6dr\") pod \"kserve-controller-manager-d6f4b65b8-grtpb\" (UID: \"6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-grtpb"
Sep 17 03:24:59.004687 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:24:59.004171    2557 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 03:24:59.004687 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:24:59.004252    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8-cert podName:0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8 nodeName:}" failed. No retries permitted until 2026-09-17 03:24:59.504231705 +0000 UTC m=+350.107327703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8-cert") pod "llmisvc-controller-manager-777c6bd7d5-tcntt" (UID: "0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8") : secret "llmisvc-webhook-server-cert" not found
Sep 17 03:24:59.004687 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:24:59.004461    2557 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 17 03:24:59.004687 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:24:59.004498    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7-cert podName:6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7 nodeName:}" failed. No retries permitted until 2026-09-17 03:24:59.504486098 +0000 UTC m=+350.107582095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7-cert") pod "kserve-controller-manager-d6f4b65b8-grtpb" (UID: "6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7") : secret "kserve-webhook-server-cert" not found
Sep 17 03:24:59.034262 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:59.034205    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvhkl\" (UniqueName: \"kubernetes.io/projected/0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8-kube-api-access-xvhkl\") pod \"llmisvc-controller-manager-777c6bd7d5-tcntt\" (UID: \"0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt"
Sep 17 03:24:59.035605 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:59.035562    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx6dr\" (UniqueName: \"kubernetes.io/projected/6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7-kube-api-access-lx6dr\") pod \"kserve-controller-manager-d6f4b65b8-grtpb\" (UID: \"6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-grtpb"
Sep 17 03:24:59.509757 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:59.509722    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7-cert\") pod \"kserve-controller-manager-d6f4b65b8-grtpb\" (UID: \"6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-grtpb"
Sep 17 03:24:59.510211 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:59.509807    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tcntt\" (UID: \"0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt"
Sep 17 03:24:59.512336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:59.512309    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tcntt\" (UID: \"0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt"
Sep 17 03:24:59.512452 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:59.512434    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7-cert\") pod \"kserve-controller-manager-d6f4b65b8-grtpb\" (UID: \"6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-grtpb"
Sep 17 03:24:59.740351 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:59.740312    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-grtpb"
Sep 17 03:24:59.754572 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:24:59.754549    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt"
Sep 17 03:25:00.866489 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:00.866458    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-grtpb"]
Sep 17 03:25:00.868036 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:00.868010    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt"]
Sep 17 03:25:00.868592 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:25:00.868562    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebefd1f_5f36_41ec_9fd3_ee0020d91fe7.slice/crio-561a6c486792286a92eb5e23b8a189c3586d63047da45085413fb75e332d29b4 WatchSource:0}: Error finding container 561a6c486792286a92eb5e23b8a189c3586d63047da45085413fb75e332d29b4: Status 404 returned error can't find the container with id 561a6c486792286a92eb5e23b8a189c3586d63047da45085413fb75e332d29b4
Sep 17 03:25:00.869465 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:25:00.869439    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0c369181_ccd7_4e5c_ab2e_bd3bbf87f0a8.slice/crio-2450c5f7a76bee2031d48fa4b179a7e2d36bc4ff930d6bcf3384d4b70121b8a2 WatchSource:0}: Error finding container 2450c5f7a76bee2031d48fa4b179a7e2d36bc4ff930d6bcf3384d4b70121b8a2: Status 404 returned error can't find the container with id 2450c5f7a76bee2031d48fa4b179a7e2d36bc4ff930d6bcf3384d4b70121b8a2
Sep 17 03:25:01.461496 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:01.461455    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd" event={"ID":"b33ba81b-b815-434c-a131-6e57c834c613","Type":"ContainerStarted","Data":"72d2d160ab0ae9ca83cf385c5f9593294482132540b3f78f11c13cdf3793ae94"}
Sep 17 03:25:01.461683 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:01.461527    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:25:01.462810 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:01.462780    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-grtpb" event={"ID":"6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7","Type":"ContainerStarted","Data":"561a6c486792286a92eb5e23b8a189c3586d63047da45085413fb75e332d29b4"}
Sep 17 03:25:01.464007 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:01.463981    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt" event={"ID":"0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8","Type":"ContainerStarted","Data":"2450c5f7a76bee2031d48fa4b179a7e2d36bc4ff930d6bcf3384d4b70121b8a2"}
Sep 17 03:25:01.528375 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:01.528337    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd" podStartSLOduration=1.928882225 podStartE2EDuration="4.528322284s" totalImagesPullingTime="2.599440059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:25:01.527331142 +0000 UTC m=+352.130427146" watchObservedRunningTime="2026-09-17 03:25:01.528322284 +0000 UTC m=+352.131418287"
Sep 17 03:25:05.491349 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:05.491313    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-grtpb" event={"ID":"6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7","Type":"ContainerStarted","Data":"56261d1472a3efbd13326f4a838f4b901fb3a592c2572b83f40e4682a65f8d2c"}
Sep 17 03:25:05.491816 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:05.491408    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-grtpb"
Sep 17 03:25:05.492920 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:05.492898    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt" event={"ID":"0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8","Type":"ContainerStarted","Data":"451ea2462e28bb3757cc93c2dc5f2846e84374b7b8cd58c028f790aa68cbbfb2"}
Sep 17 03:25:05.493039 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:05.493027    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt"
Sep 17 03:25:05.514619 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:05.514572    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-grtpb" podStartSLOduration=3.903951825 podStartE2EDuration="7.514559981s" totalImagesPullingTime="3.610608156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:24:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:25:05.513885647 +0000 UTC m=+356.116981650" watchObservedRunningTime="2026-09-17 03:25:05.514559981 +0000 UTC m=+356.117655985"
Sep 17 03:25:05.536326 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:05.536287    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt" podStartSLOduration=3.890683737 podStartE2EDuration="7.53627511s" totalImagesPullingTime="3.645591373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:24:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:25:05.534943384 +0000 UTC m=+356.138039393" watchObservedRunningTime="2026-09-17 03:25:05.53627511 +0000 UTC m=+356.139371114"
Sep 17 03:25:12.472581 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:12.472542    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-55d557bdbb-lfwgd"
Sep 17 03:25:36.500173 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:36.500140    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tcntt"
Sep 17 03:25:36.503503 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:25:36.503477    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-grtpb"
Sep 17 03:26:09.215348 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:09.215312    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"]
Sep 17 03:26:09.218849 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:09.218830    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"
Sep 17 03:26:09.221132 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:09.221106    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 03:26:09.221294 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:09.221158    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wvpck\""
Sep 17 03:26:09.221751 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:09.221735    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 03:26:09.228559 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:09.228534    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"]
Sep 17 03:26:09.321635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:09.321613    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x58j5\" (UniqueName: \"kubernetes.io/projected/7e20fa1e-e204-4967-ae4a-d443430cbcf9-kube-api-access-x58j5\") pod \"limitador-operator-controller-manager-74495c69f8-qx5wr\" (UID: \"7e20fa1e-e204-4967-ae4a-d443430cbcf9\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"
Sep 17 03:26:09.421941 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:09.421916    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x58j5\" (UniqueName: \"kubernetes.io/projected/7e20fa1e-e204-4967-ae4a-d443430cbcf9-kube-api-access-x58j5\") pod \"limitador-operator-controller-manager-74495c69f8-qx5wr\" (UID: \"7e20fa1e-e204-4967-ae4a-d443430cbcf9\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"
Sep 17 03:26:09.437459 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:09.437436    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x58j5\" (UniqueName: \"kubernetes.io/projected/7e20fa1e-e204-4967-ae4a-d443430cbcf9-kube-api-access-x58j5\") pod \"limitador-operator-controller-manager-74495c69f8-qx5wr\" (UID: \"7e20fa1e-e204-4967-ae4a-d443430cbcf9\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"
Sep 17 03:26:09.529936 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:09.529886    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"
Sep 17 03:26:09.800342 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:09.800320    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"]
Sep 17 03:26:09.802661 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:26:09.802634    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e20fa1e_e204_4967_ae4a_d443430cbcf9.slice/crio-db36344b003e5927d249bb2f11ad5424abd9c9c8e5d71c464d31911cb7672c08 WatchSource:0}: Error finding container db36344b003e5927d249bb2f11ad5424abd9c9c8e5d71c464d31911cb7672c08: Status 404 returned error can't find the container with id db36344b003e5927d249bb2f11ad5424abd9c9c8e5d71c464d31911cb7672c08
Sep 17 03:26:10.798983 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:10.798940    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr" event={"ID":"7e20fa1e-e204-4967-ae4a-d443430cbcf9","Type":"ContainerStarted","Data":"db36344b003e5927d249bb2f11ad5424abd9c9c8e5d71c464d31911cb7672c08"}
Sep 17 03:26:12.810781 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:12.810746    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr" event={"ID":"7e20fa1e-e204-4967-ae4a-d443430cbcf9","Type":"ContainerStarted","Data":"298606bcdb049b5d3573df92ab01ea557a6bd10aad6e075f7042ef4d98a0eda0"}
Sep 17 03:26:12.811127 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:12.810845    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"
Sep 17 03:26:12.830450 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:12.830407    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr" podStartSLOduration=1.341000242 podStartE2EDuration="3.830391874s" totalImagesPullingTime="2.489391632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:26:12.829243159 +0000 UTC m=+423.432339159" watchObservedRunningTime="2026-09-17 03:26:12.830391874 +0000 UTC m=+423.433487877"
Sep 17 03:26:15.569544 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:15.569508    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wh8n"]
Sep 17 03:26:15.573047 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:15.573024    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wh8n"
Sep 17 03:26:15.576974 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:15.576951    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 17 03:26:15.577059 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:15.576993    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2r8wm\""
Sep 17 03:26:15.583526 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:15.583507    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wh8n"]
Sep 17 03:26:15.670610 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:15.670583    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qrvj\" (UniqueName: \"kubernetes.io/projected/8728ba23-f898-4af8-800d-13fc061c92c4-kube-api-access-9qrvj\") pod \"dns-operator-controller-manager-7b84cd7c9c-8wh8n\" (UID: \"8728ba23-f898-4af8-800d-13fc061c92c4\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wh8n"
Sep 17 03:26:15.771021 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:15.770996    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qrvj\" (UniqueName: \"kubernetes.io/projected/8728ba23-f898-4af8-800d-13fc061c92c4-kube-api-access-9qrvj\") pod \"dns-operator-controller-manager-7b84cd7c9c-8wh8n\" (UID: \"8728ba23-f898-4af8-800d-13fc061c92c4\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wh8n"
Sep 17 03:26:15.780977 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:15.780944    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qrvj\" (UniqueName: \"kubernetes.io/projected/8728ba23-f898-4af8-800d-13fc061c92c4-kube-api-access-9qrvj\") pod \"dns-operator-controller-manager-7b84cd7c9c-8wh8n\" (UID: \"8728ba23-f898-4af8-800d-13fc061c92c4\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wh8n"
Sep 17 03:26:15.884433 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:15.884401    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wh8n"
Sep 17 03:26:16.177932 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:16.177905    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wh8n"]
Sep 17 03:26:16.179833 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:26:16.179807    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8728ba23_f898_4af8_800d_13fc061c92c4.slice/crio-b70043587bf146433bef54f61adc8ba0aab5a1f5fbffcdfa4e0505d79059c4bd WatchSource:0}: Error finding container b70043587bf146433bef54f61adc8ba0aab5a1f5fbffcdfa4e0505d79059c4bd: Status 404 returned error can't find the container with id b70043587bf146433bef54f61adc8ba0aab5a1f5fbffcdfa4e0505d79059c4bd
Sep 17 03:26:16.830927 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:16.830891    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wh8n" event={"ID":"8728ba23-f898-4af8-800d-13fc061c92c4","Type":"ContainerStarted","Data":"b70043587bf146433bef54f61adc8ba0aab5a1f5fbffcdfa4e0505d79059c4bd"}
Sep 17 03:26:18.842127 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:18.842085    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wh8n" event={"ID":"8728ba23-f898-4af8-800d-13fc061c92c4","Type":"ContainerStarted","Data":"f16c978943c24b1c10638c750edca8ea39c3a578840e59fcdc618cf85025f958"}
Sep 17 03:26:18.842651 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:18.842324    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wh8n"
Sep 17 03:26:18.860276 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:18.860224    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wh8n" podStartSLOduration=2.162293824 podStartE2EDuration="3.860172972s" totalImagesPullingTime="1.697879148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:26:18.858952422 +0000 UTC m=+429.462048426" watchObservedRunningTime="2026-09-17 03:26:18.860172972 +0000 UTC m=+429.463268976"
Sep 17 03:26:23.817617 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:23.817576    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"
Sep 17 03:26:29.849312 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:29.849264    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-8wh8n"
Sep 17 03:26:30.744615 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:30.744576    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"]
Sep 17 03:26:30.751319 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:30.751289    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"
Sep 17 03:26:30.756852 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:30.756824    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2tz2l\""
Sep 17 03:26:30.762802 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:30.762775    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"]
Sep 17 03:26:30.788000 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:30.787976    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a0865de-f01a-425f-ace5-681fbfbb79fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-grhl6\" (UID: \"9a0865de-f01a-425f-ace5-681fbfbb79fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"
Sep 17 03:26:30.788116 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:30.788033    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqckj\" (UniqueName: \"kubernetes.io/projected/9a0865de-f01a-425f-ace5-681fbfbb79fa-kube-api-access-gqckj\") pod \"kuadrant-operator-controller-manager-5f685c496f-grhl6\" (UID: \"9a0865de-f01a-425f-ace5-681fbfbb79fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"
Sep 17 03:26:30.888459 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:30.888430    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a0865de-f01a-425f-ace5-681fbfbb79fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-grhl6\" (UID: \"9a0865de-f01a-425f-ace5-681fbfbb79fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"
Sep 17 03:26:30.888847 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:30.888473    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqckj\" (UniqueName: \"kubernetes.io/projected/9a0865de-f01a-425f-ace5-681fbfbb79fa-kube-api-access-gqckj\") pod \"kuadrant-operator-controller-manager-5f685c496f-grhl6\" (UID: \"9a0865de-f01a-425f-ace5-681fbfbb79fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"
Sep 17 03:26:30.888917 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:30.888866    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a0865de-f01a-425f-ace5-681fbfbb79fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-grhl6\" (UID: \"9a0865de-f01a-425f-ace5-681fbfbb79fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"
Sep 17 03:26:30.899113 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:30.899082    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqckj\" (UniqueName: \"kubernetes.io/projected/9a0865de-f01a-425f-ace5-681fbfbb79fa-kube-api-access-gqckj\") pod \"kuadrant-operator-controller-manager-5f685c496f-grhl6\" (UID: \"9a0865de-f01a-425f-ace5-681fbfbb79fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"
Sep 17 03:26:31.062574 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.062520    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"
Sep 17 03:26:31.345449 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.342382    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"]
Sep 17 03:26:31.349090 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:26:31.349051    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0865de_f01a_425f_ace5_681fbfbb79fa.slice/crio-45409aefedb9d0d8a93d9a0e339971044c143d8da3a1a2c1a08beede717e0f16 WatchSource:0}: Error finding container 45409aefedb9d0d8a93d9a0e339971044c143d8da3a1a2c1a08beede717e0f16: Status 404 returned error can't find the container with id 45409aefedb9d0d8a93d9a0e339971044c143d8da3a1a2c1a08beede717e0f16
Sep 17 03:26:31.540571 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.540543    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"]
Sep 17 03:26:31.548755 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.548730    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"]
Sep 17 03:26:31.586710 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.586688    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"]
Sep 17 03:26:31.592276 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.592257    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"]
Sep 17 03:26:31.592381 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.592369    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"
Sep 17 03:26:31.592447 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.592424    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr" podUID="7e20fa1e-e204-4967-ae4a-d443430cbcf9" containerName="manager" containerID="cri-o://298606bcdb049b5d3573df92ab01ea557a6bd10aad6e075f7042ef4d98a0eda0" gracePeriod=2
Sep 17 03:26:31.607862 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.607838    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"]
Sep 17 03:26:31.610491 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.610310    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"]
Sep 17 03:26:31.629759 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:26:31.629732    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qx5wr\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="7e20fa1e-e204-4967-ae4a-d443430cbcf9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"
Sep 17 03:26:31.631388 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:26:31.631361    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qx5wr\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="7e20fa1e-e204-4967-ae4a-d443430cbcf9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"
Sep 17 03:26:31.695187 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.695163    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0287f1-089d-45fe-b3ad-a09ea34f1863-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-n4vfn\" (UID: \"eb0287f1-089d-45fe-b3ad-a09ea34f1863\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"
Sep 17 03:26:31.695307 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.695220    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsv8p\" (UniqueName: \"kubernetes.io/projected/eb0287f1-089d-45fe-b3ad-a09ea34f1863-kube-api-access-qsv8p\") pod \"kuadrant-operator-controller-manager-5f685c496f-n4vfn\" (UID: \"eb0287f1-089d-45fe-b3ad-a09ea34f1863\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"
Sep 17 03:26:31.796159 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.796125    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0287f1-089d-45fe-b3ad-a09ea34f1863-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-n4vfn\" (UID: \"eb0287f1-089d-45fe-b3ad-a09ea34f1863\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"
Sep 17 03:26:31.796304 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.796207    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsv8p\" (UniqueName: \"kubernetes.io/projected/eb0287f1-089d-45fe-b3ad-a09ea34f1863-kube-api-access-qsv8p\") pod \"kuadrant-operator-controller-manager-5f685c496f-n4vfn\" (UID: \"eb0287f1-089d-45fe-b3ad-a09ea34f1863\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"
Sep 17 03:26:31.796619 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.796598    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0287f1-089d-45fe-b3ad-a09ea34f1863-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-n4vfn\" (UID: \"eb0287f1-089d-45fe-b3ad-a09ea34f1863\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"
Sep 17 03:26:31.807664 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.807640    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsv8p\" (UniqueName: \"kubernetes.io/projected/eb0287f1-089d-45fe-b3ad-a09ea34f1863-kube-api-access-qsv8p\") pod \"kuadrant-operator-controller-manager-5f685c496f-n4vfn\" (UID: \"eb0287f1-089d-45fe-b3ad-a09ea34f1863\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"
Sep 17 03:26:31.813951 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.813932    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"
Sep 17 03:26:31.815716 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:26:31.815693    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qx5wr\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="7e20fa1e-e204-4967-ae4a-d443430cbcf9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"
Sep 17 03:26:31.897151 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.897129    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x58j5\" (UniqueName: \"kubernetes.io/projected/7e20fa1e-e204-4967-ae4a-d443430cbcf9-kube-api-access-x58j5\") pod \"7e20fa1e-e204-4967-ae4a-d443430cbcf9\" (UID: \"7e20fa1e-e204-4967-ae4a-d443430cbcf9\") "
Sep 17 03:26:31.899009 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.898988    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e20fa1e-e204-4967-ae4a-d443430cbcf9-kube-api-access-x58j5" (OuterVolumeSpecName: "kube-api-access-x58j5") pod "7e20fa1e-e204-4967-ae4a-d443430cbcf9" (UID: "7e20fa1e-e204-4967-ae4a-d443430cbcf9"). InnerVolumeSpecName "kube-api-access-x58j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:26:31.913178 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.912989    2557 generic.go:356] "Generic (PLEG): container finished" podID="7e20fa1e-e204-4967-ae4a-d443430cbcf9" containerID="298606bcdb049b5d3573df92ab01ea557a6bd10aad6e075f7042ef4d98a0eda0" exitCode=0
Sep 17 03:26:31.913178 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.913100    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"
Sep 17 03:26:31.915703 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:26:31.915640    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qx5wr\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="7e20fa1e-e204-4967-ae4a-d443430cbcf9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qx5wr"
Sep 17 03:26:31.918067 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.918042    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e20fa1e-e204-4967-ae4a-d443430cbcf9" path="/var/lib/kubelet/pods/7e20fa1e-e204-4967-ae4a-d443430cbcf9/volumes"
Sep 17 03:26:31.918548 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.918527    2557 scope.go:117] "RemoveContainer" containerID="298606bcdb049b5d3573df92ab01ea557a6bd10aad6e075f7042ef4d98a0eda0"
Sep 17 03:26:31.929612 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.929595    2557 scope.go:117] "RemoveContainer" containerID="298606bcdb049b5d3573df92ab01ea557a6bd10aad6e075f7042ef4d98a0eda0"
Sep 17 03:26:31.929900 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:26:31.929878    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298606bcdb049b5d3573df92ab01ea557a6bd10aad6e075f7042ef4d98a0eda0\": container with ID starting with 298606bcdb049b5d3573df92ab01ea557a6bd10aad6e075f7042ef4d98a0eda0 not found: ID does not exist" containerID="298606bcdb049b5d3573df92ab01ea557a6bd10aad6e075f7042ef4d98a0eda0"
Sep 17 03:26:31.929979 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.929914    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298606bcdb049b5d3573df92ab01ea557a6bd10aad6e075f7042ef4d98a0eda0"} err="failed to get container status \"298606bcdb049b5d3573df92ab01ea557a6bd10aad6e075f7042ef4d98a0eda0\": rpc error: code = NotFound desc = could not find container \"298606bcdb049b5d3573df92ab01ea557a6bd10aad6e075f7042ef4d98a0eda0\": container with ID starting with 298606bcdb049b5d3573df92ab01ea557a6bd10aad6e075f7042ef4d98a0eda0 not found: ID does not exist"
Sep 17 03:26:31.964063 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.964042    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"
Sep 17 03:26:31.998115 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:31.998091    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x58j5\" (UniqueName: \"kubernetes.io/projected/7e20fa1e-e204-4967-ae4a-d443430cbcf9-kube-api-access-x58j5\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:26:32.253776 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:32.253530    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"]
Sep 17 03:26:32.256831 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:26:32.256763    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0287f1_089d_45fe_b3ad_a09ea34f1863.slice/crio-12b881bbe39b3a9ff5bc921b4ecefdbb8141a7c65e318236e1e28f7e582c5d96 WatchSource:0}: Error finding container 12b881bbe39b3a9ff5bc921b4ecefdbb8141a7c65e318236e1e28f7e582c5d96: Status 404 returned error can't find the container with id 12b881bbe39b3a9ff5bc921b4ecefdbb8141a7c65e318236e1e28f7e582c5d96
Sep 17 03:26:32.923610 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:32.923575    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn" event={"ID":"eb0287f1-089d-45fe-b3ad-a09ea34f1863","Type":"ContainerStarted","Data":"12b881bbe39b3a9ff5bc921b4ecefdbb8141a7c65e318236e1e28f7e582c5d96"}
Sep 17 03:26:35.943800 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:35.943766    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn" event={"ID":"eb0287f1-089d-45fe-b3ad-a09ea34f1863","Type":"ContainerStarted","Data":"fcb37a44b2386e46c642aabaf61dfefc3ef39aa946d759e31a218ab280d732fc"}
Sep 17 03:26:35.944226 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:35.943881    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"
Sep 17 03:26:35.945482 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:35.945438    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6" podUID="9a0865de-f01a-425f-ace5-681fbfbb79fa" containerName="manager" containerID="cri-o://12aa92a3984aec3d7278a6fd88dcc93b04edf7f977daa57a16656ad86dc78f02" gracePeriod=2
Sep 17 03:26:35.963968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:35.963929    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn" podStartSLOduration=2.110083304 podStartE2EDuration="4.963915985s" totalImagesPullingTime="2.853832681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:26:35.961622111 +0000 UTC m=+446.564718137" watchObservedRunningTime="2026-09-17 03:26:35.963915985 +0000 UTC m=+446.567011989"
Sep 17 03:26:36.180303 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:36.180277    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"
Sep 17 03:26:36.182150 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:26:36.182122    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-grhl6\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="9a0865de-f01a-425f-ace5-681fbfbb79fa" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"
Sep 17 03:26:36.237617 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:36.237559    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqckj\" (UniqueName: \"kubernetes.io/projected/9a0865de-f01a-425f-ace5-681fbfbb79fa-kube-api-access-gqckj\") pod \"9a0865de-f01a-425f-ace5-681fbfbb79fa\" (UID: \"9a0865de-f01a-425f-ace5-681fbfbb79fa\") "
Sep 17 03:26:36.237617 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:36.237613    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a0865de-f01a-425f-ace5-681fbfbb79fa-extensions-socket-volume\") pod \"9a0865de-f01a-425f-ace5-681fbfbb79fa\" (UID: \"9a0865de-f01a-425f-ace5-681fbfbb79fa\") "
Sep 17 03:26:36.237881 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:36.237858    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0865de-f01a-425f-ace5-681fbfbb79fa-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9a0865de-f01a-425f-ace5-681fbfbb79fa" (UID: "9a0865de-f01a-425f-ace5-681fbfbb79fa"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:26:36.239420 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:36.239402    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0865de-f01a-425f-ace5-681fbfbb79fa-kube-api-access-gqckj" (OuterVolumeSpecName: "kube-api-access-gqckj") pod "9a0865de-f01a-425f-ace5-681fbfbb79fa" (UID: "9a0865de-f01a-425f-ace5-681fbfbb79fa"). InnerVolumeSpecName "kube-api-access-gqckj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:26:36.338322 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:36.338297    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqckj\" (UniqueName: \"kubernetes.io/projected/9a0865de-f01a-425f-ace5-681fbfbb79fa-kube-api-access-gqckj\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:26:36.338322 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:36.338320    2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a0865de-f01a-425f-ace5-681fbfbb79fa-extensions-socket-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:26:36.952207 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:36.952172    2557 generic.go:356] "Generic (PLEG): container finished" podID="9a0865de-f01a-425f-ace5-681fbfbb79fa" containerID="12aa92a3984aec3d7278a6fd88dcc93b04edf7f977daa57a16656ad86dc78f02" exitCode=2
Sep 17 03:26:36.952667 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:36.952239    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"
Sep 17 03:26:36.952667 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:36.952276    2557 scope.go:117] "RemoveContainer" containerID="12aa92a3984aec3d7278a6fd88dcc93b04edf7f977daa57a16656ad86dc78f02"
Sep 17 03:26:36.954297 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:26:36.954259    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-grhl6\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="9a0865de-f01a-425f-ace5-681fbfbb79fa" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"
Sep 17 03:26:36.961181 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:36.961154    2557 scope.go:117] "RemoveContainer" containerID="12aa92a3984aec3d7278a6fd88dcc93b04edf7f977daa57a16656ad86dc78f02"
Sep 17 03:26:36.961492 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:26:36.961475    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12aa92a3984aec3d7278a6fd88dcc93b04edf7f977daa57a16656ad86dc78f02\": container with ID starting with 12aa92a3984aec3d7278a6fd88dcc93b04edf7f977daa57a16656ad86dc78f02 not found: ID does not exist" containerID="12aa92a3984aec3d7278a6fd88dcc93b04edf7f977daa57a16656ad86dc78f02"
Sep 17 03:26:36.961547 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:36.961500    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12aa92a3984aec3d7278a6fd88dcc93b04edf7f977daa57a16656ad86dc78f02"} err="failed to get container status \"12aa92a3984aec3d7278a6fd88dcc93b04edf7f977daa57a16656ad86dc78f02\": rpc error: code = NotFound desc = could not find container \"12aa92a3984aec3d7278a6fd88dcc93b04edf7f977daa57a16656ad86dc78f02\": container with ID starting with 12aa92a3984aec3d7278a6fd88dcc93b04edf7f977daa57a16656ad86dc78f02 not found: ID does not exist"
Sep 17 03:26:36.962474 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:26:36.962449    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-grhl6\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="9a0865de-f01a-425f-ace5-681fbfbb79fa" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-grhl6"
Sep 17 03:26:37.915107 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:37.915062    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a0865de-f01a-425f-ace5-681fbfbb79fa" path="/var/lib/kubelet/pods/9a0865de-f01a-425f-ace5-681fbfbb79fa/volumes"
Sep 17 03:26:46.955010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:46.954975    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"
Sep 17 03:26:59.457240 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:59.457186    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"]
Sep 17 03:26:59.457705 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:59.457512    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn" podUID="eb0287f1-089d-45fe-b3ad-a09ea34f1863" containerName="manager" containerID="cri-o://fcb37a44b2386e46c642aabaf61dfefc3ef39aa946d759e31a218ab280d732fc" gracePeriod=10
Sep 17 03:26:59.704202 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:59.704168    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"
Sep 17 03:26:59.827534 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:59.827465    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0287f1-089d-45fe-b3ad-a09ea34f1863-extensions-socket-volume\") pod \"eb0287f1-089d-45fe-b3ad-a09ea34f1863\" (UID: \"eb0287f1-089d-45fe-b3ad-a09ea34f1863\") "
Sep 17 03:26:59.827534 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:59.827508    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsv8p\" (UniqueName: \"kubernetes.io/projected/eb0287f1-089d-45fe-b3ad-a09ea34f1863-kube-api-access-qsv8p\") pod \"eb0287f1-089d-45fe-b3ad-a09ea34f1863\" (UID: \"eb0287f1-089d-45fe-b3ad-a09ea34f1863\") "
Sep 17 03:26:59.827924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:59.827894    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb0287f1-089d-45fe-b3ad-a09ea34f1863-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "eb0287f1-089d-45fe-b3ad-a09ea34f1863" (UID: "eb0287f1-089d-45fe-b3ad-a09ea34f1863"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:26:59.829510 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:59.829486    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0287f1-089d-45fe-b3ad-a09ea34f1863-kube-api-access-qsv8p" (OuterVolumeSpecName: "kube-api-access-qsv8p") pod "eb0287f1-089d-45fe-b3ad-a09ea34f1863" (UID: "eb0287f1-089d-45fe-b3ad-a09ea34f1863"). InnerVolumeSpecName "kube-api-access-qsv8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:26:59.928660 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:59.928639    2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0287f1-089d-45fe-b3ad-a09ea34f1863-extensions-socket-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:26:59.928660 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:26:59.928660    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsv8p\" (UniqueName: \"kubernetes.io/projected/eb0287f1-089d-45fe-b3ad-a09ea34f1863-kube-api-access-qsv8p\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:27:00.069045 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:00.069017    2557 generic.go:356] "Generic (PLEG): container finished" podID="eb0287f1-089d-45fe-b3ad-a09ea34f1863" containerID="fcb37a44b2386e46c642aabaf61dfefc3ef39aa946d759e31a218ab280d732fc" exitCode=0
Sep 17 03:27:00.069162 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:00.069050    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn" event={"ID":"eb0287f1-089d-45fe-b3ad-a09ea34f1863","Type":"ContainerDied","Data":"fcb37a44b2386e46c642aabaf61dfefc3ef39aa946d759e31a218ab280d732fc"}
Sep 17 03:27:00.069162 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:00.069060    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"
Sep 17 03:27:00.069162 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:00.069097    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn" event={"ID":"eb0287f1-089d-45fe-b3ad-a09ea34f1863","Type":"ContainerDied","Data":"12b881bbe39b3a9ff5bc921b4ecefdbb8141a7c65e318236e1e28f7e582c5d96"}
Sep 17 03:27:00.069162 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:00.069129    2557 scope.go:117] "RemoveContainer" containerID="fcb37a44b2386e46c642aabaf61dfefc3ef39aa946d759e31a218ab280d732fc"
Sep 17 03:27:00.077050 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:00.077028    2557 scope.go:117] "RemoveContainer" containerID="fcb37a44b2386e46c642aabaf61dfefc3ef39aa946d759e31a218ab280d732fc"
Sep 17 03:27:00.077325 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:27:00.077303    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb37a44b2386e46c642aabaf61dfefc3ef39aa946d759e31a218ab280d732fc\": container with ID starting with fcb37a44b2386e46c642aabaf61dfefc3ef39aa946d759e31a218ab280d732fc not found: ID does not exist" containerID="fcb37a44b2386e46c642aabaf61dfefc3ef39aa946d759e31a218ab280d732fc"
Sep 17 03:27:00.077415 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:00.077332    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb37a44b2386e46c642aabaf61dfefc3ef39aa946d759e31a218ab280d732fc"} err="failed to get container status \"fcb37a44b2386e46c642aabaf61dfefc3ef39aa946d759e31a218ab280d732fc\": rpc error: code = NotFound desc = could not find container \"fcb37a44b2386e46c642aabaf61dfefc3ef39aa946d759e31a218ab280d732fc\": container with ID starting with fcb37a44b2386e46c642aabaf61dfefc3ef39aa946d759e31a218ab280d732fc not found: ID does not exist"
Sep 17 03:27:00.084352 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:00.084298    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"]
Sep 17 03:27:00.088634 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:00.088612    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-n4vfn"]
Sep 17 03:27:01.916252 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:01.916209    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb0287f1-089d-45fe-b3ad-a09ea34f1863" path="/var/lib/kubelet/pods/eb0287f1-089d-45fe-b3ad-a09ea34f1863/volumes"
Sep 17 03:27:03.432715 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.432683    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"]
Sep 17 03:27:03.433236 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.433217    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a0865de-f01a-425f-ace5-681fbfbb79fa" containerName="manager"
Sep 17 03:27:03.433325 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.433238    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a0865de-f01a-425f-ace5-681fbfbb79fa" containerName="manager"
Sep 17 03:27:03.433325 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.433254    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e20fa1e-e204-4967-ae4a-d443430cbcf9" containerName="manager"
Sep 17 03:27:03.433325 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.433262    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e20fa1e-e204-4967-ae4a-d443430cbcf9" containerName="manager"
Sep 17 03:27:03.433325 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.433272    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb0287f1-089d-45fe-b3ad-a09ea34f1863" containerName="manager"
Sep 17 03:27:03.433325 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.433279    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb0287f1-089d-45fe-b3ad-a09ea34f1863" containerName="manager"
Sep 17 03:27:03.433581 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.433400    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb0287f1-089d-45fe-b3ad-a09ea34f1863" containerName="manager"
Sep 17 03:27:03.433581 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.433412    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e20fa1e-e204-4967-ae4a-d443430cbcf9" containerName="manager"
Sep 17 03:27:03.433581 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.433424    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a0865de-f01a-425f-ace5-681fbfbb79fa" containerName="manager"
Sep 17 03:27:03.436356 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.436328    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.438660 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.438642    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wszjv\""
Sep 17 03:27:03.449642 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.449623    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"]
Sep 17 03:27:03.555295 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.555273    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a4d22e-eade-4378-bdc4-5f42eb080071-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.555405 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.555304    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a4d22e-eade-4378-bdc4-5f42eb080071-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.555405 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.555352    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a4d22e-eade-4378-bdc4-5f42eb080071-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.555500 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.555406    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a4d22e-eade-4378-bdc4-5f42eb080071-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.555500 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.555432    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a4d22e-eade-4378-bdc4-5f42eb080071-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.555500 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.555453    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a4d22e-eade-4378-bdc4-5f42eb080071-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.555500 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.555474    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a4d22e-eade-4378-bdc4-5f42eb080071-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.555683 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.555517    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a4d22e-eade-4378-bdc4-5f42eb080071-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.555683 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.555581    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbxw2\" (UniqueName: \"kubernetes.io/projected/f1a4d22e-eade-4378-bdc4-5f42eb080071-kube-api-access-fbxw2\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.656317 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.656295    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a4d22e-eade-4378-bdc4-5f42eb080071-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.656454 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.656325    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a4d22e-eade-4378-bdc4-5f42eb080071-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.656454 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.656355    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a4d22e-eade-4378-bdc4-5f42eb080071-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.656454 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.656378    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a4d22e-eade-4378-bdc4-5f42eb080071-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.656454 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.656424    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a4d22e-eade-4378-bdc4-5f42eb080071-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.656677 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.656472    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a4d22e-eade-4378-bdc4-5f42eb080071-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.656677 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.656516    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxw2\" (UniqueName: \"kubernetes.io/projected/f1a4d22e-eade-4378-bdc4-5f42eb080071-kube-api-access-fbxw2\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.656677 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.656548    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a4d22e-eade-4378-bdc4-5f42eb080071-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.656677 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.656576    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a4d22e-eade-4378-bdc4-5f42eb080071-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.656903 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.656772    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1a4d22e-eade-4378-bdc4-5f42eb080071-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.656903 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.656822    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a4d22e-eade-4378-bdc4-5f42eb080071-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.657005 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.656984    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1a4d22e-eade-4378-bdc4-5f42eb080071-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.657269 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.657239    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1a4d22e-eade-4378-bdc4-5f42eb080071-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.657269 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.657256    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1a4d22e-eade-4378-bdc4-5f42eb080071-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.658790 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.658763    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1a4d22e-eade-4378-bdc4-5f42eb080071-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.658975 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.658959    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1a4d22e-eade-4378-bdc4-5f42eb080071-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.664544 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.664521    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1a4d22e-eade-4378-bdc4-5f42eb080071-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.664625 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.664589    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbxw2\" (UniqueName: \"kubernetes.io/projected/f1a4d22e-eade-4378-bdc4-5f42eb080071-kube-api-access-fbxw2\") pod \"maas-default-gateway-openshift-default-78bb78ff78-q9jxx\" (UID: \"f1a4d22e-eade-4378-bdc4-5f42eb080071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:03.745706 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:03.745658    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:04.023678 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:04.023650    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"]
Sep 17 03:27:04.025278 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:27:04.025251    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a4d22e_eade_4378_bdc4_5f42eb080071.slice/crio-14b59d52c22ac62f4a99c1a5000818b55176315b86bfae767467a88ad4604cfc WatchSource:0}: Error finding container 14b59d52c22ac62f4a99c1a5000818b55176315b86bfae767467a88ad4604cfc: Status 404 returned error can't find the container with id 14b59d52c22ac62f4a99c1a5000818b55176315b86bfae767467a88ad4604cfc
Sep 17 03:27:04.027282 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:04.027250    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:27:04.027347 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:04.027314    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:27:04.027347 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:04.027342    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:27:04.092300 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:04.092274    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx" event={"ID":"f1a4d22e-eade-4378-bdc4-5f42eb080071","Type":"ContainerStarted","Data":"68e83da339405be096476a6bc528ba6d43ad79be7e7680af2e2e9392a8a23d88"}
Sep 17 03:27:04.092392 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:04.092324    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx" event={"ID":"f1a4d22e-eade-4378-bdc4-5f42eb080071","Type":"ContainerStarted","Data":"14b59d52c22ac62f4a99c1a5000818b55176315b86bfae767467a88ad4604cfc"}
Sep 17 03:27:05.133995 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:05.133943    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx" podStartSLOduration=2.13392736 podStartE2EDuration="2.13392736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:27:05.132747859 +0000 UTC m=+475.735843864" watchObservedRunningTime="2026-09-17 03:27:05.13392736 +0000 UTC m=+475.737023364"
Sep 17 03:27:05.746430 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:05.746397    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:05.751213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:05.751167    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:06.102768 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:06.102684    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:06.103729 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:06.103706    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:27:17.046302 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.046261    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5j57s"]
Sep 17 03:27:17.050767 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.050741    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5j57s"
Sep 17 03:27:17.053699 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.053672    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 03:27:17.053854 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.053765    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f5jg8\""
Sep 17 03:27:17.072560 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.072528    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5j57s"]
Sep 17 03:27:17.145887 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.145853    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5j57s"]
Sep 17 03:27:17.166617 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.166589    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d03efe67-d6bb-482c-b703-61a1416f0a6f-config-file\") pod \"limitador-limitador-8588749977-5j57s\" (UID: \"d03efe67-d6bb-482c-b703-61a1416f0a6f\") " pod="kuadrant-system/limitador-limitador-8588749977-5j57s"
Sep 17 03:27:17.166752 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.166674    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slnmx\" (UniqueName: \"kubernetes.io/projected/d03efe67-d6bb-482c-b703-61a1416f0a6f-kube-api-access-slnmx\") pod \"limitador-limitador-8588749977-5j57s\" (UID: \"d03efe67-d6bb-482c-b703-61a1416f0a6f\") " pod="kuadrant-system/limitador-limitador-8588749977-5j57s"
Sep 17 03:27:17.267478 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.267449    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d03efe67-d6bb-482c-b703-61a1416f0a6f-config-file\") pod \"limitador-limitador-8588749977-5j57s\" (UID: \"d03efe67-d6bb-482c-b703-61a1416f0a6f\") " pod="kuadrant-system/limitador-limitador-8588749977-5j57s"
Sep 17 03:27:17.267631 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.267520    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slnmx\" (UniqueName: \"kubernetes.io/projected/d03efe67-d6bb-482c-b703-61a1416f0a6f-kube-api-access-slnmx\") pod \"limitador-limitador-8588749977-5j57s\" (UID: \"d03efe67-d6bb-482c-b703-61a1416f0a6f\") " pod="kuadrant-system/limitador-limitador-8588749977-5j57s"
Sep 17 03:27:17.268030 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.268012    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d03efe67-d6bb-482c-b703-61a1416f0a6f-config-file\") pod \"limitador-limitador-8588749977-5j57s\" (UID: \"d03efe67-d6bb-482c-b703-61a1416f0a6f\") " pod="kuadrant-system/limitador-limitador-8588749977-5j57s"
Sep 17 03:27:17.275830 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.275804    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slnmx\" (UniqueName: \"kubernetes.io/projected/d03efe67-d6bb-482c-b703-61a1416f0a6f-kube-api-access-slnmx\") pod \"limitador-limitador-8588749977-5j57s\" (UID: \"d03efe67-d6bb-482c-b703-61a1416f0a6f\") " pod="kuadrant-system/limitador-limitador-8588749977-5j57s"
Sep 17 03:27:17.368658 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.368636    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5j57s"
Sep 17 03:27:17.640918 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.640845    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5j57s"]
Sep 17 03:27:17.644258 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:27:17.644231    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03efe67_d6bb_482c_b703_61a1416f0a6f.slice/crio-46afdf551fe9c38887ea061dfa52463c044caca9ce9b3387e64c945f6d5f7921 WatchSource:0}: Error finding container 46afdf551fe9c38887ea061dfa52463c044caca9ce9b3387e64c945f6d5f7921: Status 404 returned error can't find the container with id 46afdf551fe9c38887ea061dfa52463c044caca9ce9b3387e64c945f6d5f7921
Sep 17 03:27:17.976866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.976811    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-d6gmw"]
Sep 17 03:27:17.981450 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.981434    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d6gmw"
Sep 17 03:27:17.983472 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.983455    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-g5wqw\""
Sep 17 03:27:17.987772 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:17.987748    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d6gmw"]
Sep 17 03:27:18.072905 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:18.072883    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq7xw\" (UniqueName: \"kubernetes.io/projected/331148a9-a72e-48fa-8609-9bbb348ee04a-kube-api-access-sq7xw\") pod \"authorino-59c974bd48-d6gmw\" (UID: \"331148a9-a72e-48fa-8609-9bbb348ee04a\") " pod="kuadrant-system/authorino-59c974bd48-d6gmw"
Sep 17 03:27:18.160854 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:18.160828    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5j57s" event={"ID":"d03efe67-d6bb-482c-b703-61a1416f0a6f","Type":"ContainerStarted","Data":"46afdf551fe9c38887ea061dfa52463c044caca9ce9b3387e64c945f6d5f7921"}
Sep 17 03:27:18.173373 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:18.173352    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq7xw\" (UniqueName: \"kubernetes.io/projected/331148a9-a72e-48fa-8609-9bbb348ee04a-kube-api-access-sq7xw\") pod \"authorino-59c974bd48-d6gmw\" (UID: \"331148a9-a72e-48fa-8609-9bbb348ee04a\") " pod="kuadrant-system/authorino-59c974bd48-d6gmw"
Sep 17 03:27:18.181632 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:18.181604    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq7xw\" (UniqueName: \"kubernetes.io/projected/331148a9-a72e-48fa-8609-9bbb348ee04a-kube-api-access-sq7xw\") pod \"authorino-59c974bd48-d6gmw\" (UID: \"331148a9-a72e-48fa-8609-9bbb348ee04a\") " pod="kuadrant-system/authorino-59c974bd48-d6gmw"
Sep 17 03:27:18.291460 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:18.291405    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d6gmw"
Sep 17 03:27:18.566786 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:18.566697    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d6gmw"]
Sep 17 03:27:18.570749 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:27:18.570717    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod331148a9_a72e_48fa_8609_9bbb348ee04a.slice/crio-cf504f9702de88da3c47ba2af1c82f377663648085cab1cad71060873a9b6066 WatchSource:0}: Error finding container cf504f9702de88da3c47ba2af1c82f377663648085cab1cad71060873a9b6066: Status 404 returned error can't find the container with id cf504f9702de88da3c47ba2af1c82f377663648085cab1cad71060873a9b6066
Sep 17 03:27:19.167541 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:19.167508    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5j57s" event={"ID":"d03efe67-d6bb-482c-b703-61a1416f0a6f","Type":"ContainerStarted","Data":"a01f69499154798b0fa4d446d8f08f77a998ddd070042191b85ba2d0c10ebee1"}
Sep 17 03:27:19.167895 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:19.167570    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-5j57s"
Sep 17 03:27:19.168804 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:19.168781    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d6gmw" event={"ID":"331148a9-a72e-48fa-8609-9bbb348ee04a","Type":"ContainerStarted","Data":"cf504f9702de88da3c47ba2af1c82f377663648085cab1cad71060873a9b6066"}
Sep 17 03:27:19.181664 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:19.181587    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-5j57s" podStartSLOduration=0.899698224 podStartE2EDuration="2.181572852s" totalImagesPullingTime="1.281874628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:27:19.181069354 +0000 UTC m=+489.784165361" watchObservedRunningTime="2026-09-17 03:27:19.181572852 +0000 UTC m=+489.784668848"
Sep 17 03:27:21.185188 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:21.185151    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d6gmw" event={"ID":"331148a9-a72e-48fa-8609-9bbb348ee04a","Type":"ContainerStarted","Data":"fa7da991aeaa75448dceb33b49140254918b649a80daf21ec8f31a831af86019"}
Sep 17 03:27:21.199981 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:21.199935    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-d6gmw" podStartSLOduration=2.3683189909999998 podStartE2EDuration="4.199921367s" totalImagesPullingTime="1.831602376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:27:21.198494184 +0000 UTC m=+491.801590189" watchObservedRunningTime="2026-09-17 03:27:21.199921367 +0000 UTC m=+491.803017372"
Sep 17 03:27:30.175250 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:30.175221    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-5j57s"
Sep 17 03:27:30.604938 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:30.604858    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5j57s"]
Sep 17 03:27:30.605086 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:30.605047    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-5j57s" podUID="d03efe67-d6bb-482c-b703-61a1416f0a6f" containerName="limitador" containerID="cri-o://a01f69499154798b0fa4d446d8f08f77a998ddd070042191b85ba2d0c10ebee1" gracePeriod=30
Sep 17 03:27:31.147287 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.147263    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5j57s"
Sep 17 03:27:31.171814 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.171757    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slnmx\" (UniqueName: \"kubernetes.io/projected/d03efe67-d6bb-482c-b703-61a1416f0a6f-kube-api-access-slnmx\") pod \"d03efe67-d6bb-482c-b703-61a1416f0a6f\" (UID: \"d03efe67-d6bb-482c-b703-61a1416f0a6f\") "
Sep 17 03:27:31.171913 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.171840    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d03efe67-d6bb-482c-b703-61a1416f0a6f-config-file\") pod \"d03efe67-d6bb-482c-b703-61a1416f0a6f\" (UID: \"d03efe67-d6bb-482c-b703-61a1416f0a6f\") "
Sep 17 03:27:31.172295 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.172263    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03efe67-d6bb-482c-b703-61a1416f0a6f-config-file" (OuterVolumeSpecName: "config-file") pod "d03efe67-d6bb-482c-b703-61a1416f0a6f" (UID: "d03efe67-d6bb-482c-b703-61a1416f0a6f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:27:31.173720 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.173688    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03efe67-d6bb-482c-b703-61a1416f0a6f-kube-api-access-slnmx" (OuterVolumeSpecName: "kube-api-access-slnmx") pod "d03efe67-d6bb-482c-b703-61a1416f0a6f" (UID: "d03efe67-d6bb-482c-b703-61a1416f0a6f"). InnerVolumeSpecName "kube-api-access-slnmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:27:31.234658 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.234627    2557 generic.go:356] "Generic (PLEG): container finished" podID="d03efe67-d6bb-482c-b703-61a1416f0a6f" containerID="a01f69499154798b0fa4d446d8f08f77a998ddd070042191b85ba2d0c10ebee1" exitCode=0
Sep 17 03:27:31.234924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.234672    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5j57s"
Sep 17 03:27:31.234924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.234686    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5j57s" event={"ID":"d03efe67-d6bb-482c-b703-61a1416f0a6f","Type":"ContainerDied","Data":"a01f69499154798b0fa4d446d8f08f77a998ddd070042191b85ba2d0c10ebee1"}
Sep 17 03:27:31.234924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.234736    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5j57s" event={"ID":"d03efe67-d6bb-482c-b703-61a1416f0a6f","Type":"ContainerDied","Data":"46afdf551fe9c38887ea061dfa52463c044caca9ce9b3387e64c945f6d5f7921"}
Sep 17 03:27:31.234924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.234767    2557 scope.go:117] "RemoveContainer" containerID="a01f69499154798b0fa4d446d8f08f77a998ddd070042191b85ba2d0c10ebee1"
Sep 17 03:27:31.243083 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.243063    2557 scope.go:117] "RemoveContainer" containerID="a01f69499154798b0fa4d446d8f08f77a998ddd070042191b85ba2d0c10ebee1"
Sep 17 03:27:31.243377 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:27:31.243359    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01f69499154798b0fa4d446d8f08f77a998ddd070042191b85ba2d0c10ebee1\": container with ID starting with a01f69499154798b0fa4d446d8f08f77a998ddd070042191b85ba2d0c10ebee1 not found: ID does not exist" containerID="a01f69499154798b0fa4d446d8f08f77a998ddd070042191b85ba2d0c10ebee1"
Sep 17 03:27:31.243426 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.243386    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01f69499154798b0fa4d446d8f08f77a998ddd070042191b85ba2d0c10ebee1"} err="failed to get container status \"a01f69499154798b0fa4d446d8f08f77a998ddd070042191b85ba2d0c10ebee1\": rpc error: code = NotFound desc = could not find container \"a01f69499154798b0fa4d446d8f08f77a998ddd070042191b85ba2d0c10ebee1\": container with ID starting with a01f69499154798b0fa4d446d8f08f77a998ddd070042191b85ba2d0c10ebee1 not found: ID does not exist"
Sep 17 03:27:31.254211 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.254168    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5j57s"]
Sep 17 03:27:31.256236 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.256218    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5j57s"]
Sep 17 03:27:31.272932 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.272911    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slnmx\" (UniqueName: \"kubernetes.io/projected/d03efe67-d6bb-482c-b703-61a1416f0a6f-kube-api-access-slnmx\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:27:31.273026 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.272933    2557 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d03efe67-d6bb-482c-b703-61a1416f0a6f-config-file\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:27:31.914836 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:31.914806    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d03efe67-d6bb-482c-b703-61a1416f0a6f" path="/var/lib/kubelet/pods/d03efe67-d6bb-482c-b703-61a1416f0a6f/volumes"
Sep 17 03:27:32.904570 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:32.904531    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5"]
Sep 17 03:27:32.904974 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:32.904906    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d03efe67-d6bb-482c-b703-61a1416f0a6f" containerName="limitador"
Sep 17 03:27:32.904974 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:32.904917    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d03efe67-d6bb-482c-b703-61a1416f0a6f" containerName="limitador"
Sep 17 03:27:32.905050 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:32.904984    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d03efe67-d6bb-482c-b703-61a1416f0a6f" containerName="limitador"
Sep 17 03:27:32.909654 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:32.909629    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5"
Sep 17 03:27:32.911854 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:32.911829    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-4lth2\""
Sep 17 03:27:32.911973 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:32.911850    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 17 03:27:32.912499 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:32.912474    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 03:27:32.912626 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:32.912544    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 17 03:27:32.923138 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:32.923110    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5"]
Sep 17 03:27:32.984637 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:32.984599    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gffgt\" (UniqueName: \"kubernetes.io/projected/078ee1f5-83c0-42a0-a1a8-012ca1341032-kube-api-access-gffgt\") pod \"postgres-69fb4594cf-jhbm5\" (UID: \"078ee1f5-83c0-42a0-a1a8-012ca1341032\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5"
Sep 17 03:27:32.984752 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:32.984719    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/078ee1f5-83c0-42a0-a1a8-012ca1341032-data\") pod \"postgres-69fb4594cf-jhbm5\" (UID: \"078ee1f5-83c0-42a0-a1a8-012ca1341032\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5"
Sep 17 03:27:33.085538 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:33.085500    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gffgt\" (UniqueName: \"kubernetes.io/projected/078ee1f5-83c0-42a0-a1a8-012ca1341032-kube-api-access-gffgt\") pod \"postgres-69fb4594cf-jhbm5\" (UID: \"078ee1f5-83c0-42a0-a1a8-012ca1341032\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5"
Sep 17 03:27:33.085676 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:33.085586    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/078ee1f5-83c0-42a0-a1a8-012ca1341032-data\") pod \"postgres-69fb4594cf-jhbm5\" (UID: \"078ee1f5-83c0-42a0-a1a8-012ca1341032\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5"
Sep 17 03:27:33.085926 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:33.085902    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/078ee1f5-83c0-42a0-a1a8-012ca1341032-data\") pod \"postgres-69fb4594cf-jhbm5\" (UID: \"078ee1f5-83c0-42a0-a1a8-012ca1341032\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5"
Sep 17 03:27:33.093931 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:33.093908    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gffgt\" (UniqueName: \"kubernetes.io/projected/078ee1f5-83c0-42a0-a1a8-012ca1341032-kube-api-access-gffgt\") pod \"postgres-69fb4594cf-jhbm5\" (UID: \"078ee1f5-83c0-42a0-a1a8-012ca1341032\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5"
Sep 17 03:27:33.222923 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:33.222853    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5"
Sep 17 03:27:33.496145 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:33.495986    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5"]
Sep 17 03:27:33.498827 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:27:33.498800    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078ee1f5_83c0_42a0_a1a8_012ca1341032.slice/crio-20c6eecb53565958af69dbdc7705621c616ad52c1ea3af2641099475ee40188d WatchSource:0}: Error finding container 20c6eecb53565958af69dbdc7705621c616ad52c1ea3af2641099475ee40188d: Status 404 returned error can't find the container with id 20c6eecb53565958af69dbdc7705621c616ad52c1ea3af2641099475ee40188d
Sep 17 03:27:34.255013 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:34.254971    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5" event={"ID":"078ee1f5-83c0-42a0-a1a8-012ca1341032","Type":"ContainerStarted","Data":"20c6eecb53565958af69dbdc7705621c616ad52c1ea3af2641099475ee40188d"}
Sep 17 03:27:39.281917 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:39.281883    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5" event={"ID":"078ee1f5-83c0-42a0-a1a8-012ca1341032","Type":"ContainerStarted","Data":"3a9c4ade2f27fab4d3eaa302834ffc48bd1995abb9818a2aab62ac69a34ca567"}
Sep 17 03:27:39.282291 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:39.281959    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5"
Sep 17 03:27:39.300244 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:39.300171    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5" podStartSLOduration=2.587377814 podStartE2EDuration="7.300153776s" totalImagesPullingTime="4.712775962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:27:39.295975251 +0000 UTC m=+509.899071253" watchObservedRunningTime="2026-09-17 03:27:39.300153776 +0000 UTC m=+509.903249781"
Sep 17 03:27:45.316474 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:45.316446    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jhbm5"
Sep 17 03:27:47.519885 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:47.519850    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d6gmw"]
Sep 17 03:27:47.520392 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:47.520093    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-d6gmw" podUID="331148a9-a72e-48fa-8609-9bbb348ee04a" containerName="authorino" containerID="cri-o://fa7da991aeaa75448dceb33b49140254918b649a80daf21ec8f31a831af86019" gracePeriod=30
Sep 17 03:27:47.762550 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:47.762525    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d6gmw"
Sep 17 03:27:47.804644 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:47.804587    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq7xw\" (UniqueName: \"kubernetes.io/projected/331148a9-a72e-48fa-8609-9bbb348ee04a-kube-api-access-sq7xw\") pod \"331148a9-a72e-48fa-8609-9bbb348ee04a\" (UID: \"331148a9-a72e-48fa-8609-9bbb348ee04a\") "
Sep 17 03:27:47.806505 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:47.806482    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331148a9-a72e-48fa-8609-9bbb348ee04a-kube-api-access-sq7xw" (OuterVolumeSpecName: "kube-api-access-sq7xw") pod "331148a9-a72e-48fa-8609-9bbb348ee04a" (UID: "331148a9-a72e-48fa-8609-9bbb348ee04a"). InnerVolumeSpecName "kube-api-access-sq7xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:27:47.905157 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:47.905133    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq7xw\" (UniqueName: \"kubernetes.io/projected/331148a9-a72e-48fa-8609-9bbb348ee04a-kube-api-access-sq7xw\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:27:48.327448 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:48.327422    2557 generic.go:356] "Generic (PLEG): container finished" podID="331148a9-a72e-48fa-8609-9bbb348ee04a" containerID="fa7da991aeaa75448dceb33b49140254918b649a80daf21ec8f31a831af86019" exitCode=0
Sep 17 03:27:48.327575 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:48.327460    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-d6gmw"
Sep 17 03:27:48.327575 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:48.327512    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d6gmw" event={"ID":"331148a9-a72e-48fa-8609-9bbb348ee04a","Type":"ContainerDied","Data":"fa7da991aeaa75448dceb33b49140254918b649a80daf21ec8f31a831af86019"}
Sep 17 03:27:48.327575 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:48.327555    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-d6gmw" event={"ID":"331148a9-a72e-48fa-8609-9bbb348ee04a","Type":"ContainerDied","Data":"cf504f9702de88da3c47ba2af1c82f377663648085cab1cad71060873a9b6066"}
Sep 17 03:27:48.327575 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:48.327573    2557 scope.go:117] "RemoveContainer" containerID="fa7da991aeaa75448dceb33b49140254918b649a80daf21ec8f31a831af86019"
Sep 17 03:27:48.336001 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:48.335984    2557 scope.go:117] "RemoveContainer" containerID="fa7da991aeaa75448dceb33b49140254918b649a80daf21ec8f31a831af86019"
Sep 17 03:27:48.336273 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:27:48.336253    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7da991aeaa75448dceb33b49140254918b649a80daf21ec8f31a831af86019\": container with ID starting with fa7da991aeaa75448dceb33b49140254918b649a80daf21ec8f31a831af86019 not found: ID does not exist" containerID="fa7da991aeaa75448dceb33b49140254918b649a80daf21ec8f31a831af86019"
Sep 17 03:27:48.336328 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:48.336279    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7da991aeaa75448dceb33b49140254918b649a80daf21ec8f31a831af86019"} err="failed to get container status \"fa7da991aeaa75448dceb33b49140254918b649a80daf21ec8f31a831af86019\": rpc error: code = NotFound desc = could not find container \"fa7da991aeaa75448dceb33b49140254918b649a80daf21ec8f31a831af86019\": container with ID starting with fa7da991aeaa75448dceb33b49140254918b649a80daf21ec8f31a831af86019 not found: ID does not exist"
Sep 17 03:27:48.341837 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:48.341815    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d6gmw"]
Sep 17 03:27:48.345953 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:48.345931    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-d6gmw"]
Sep 17 03:27:49.914479 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:49.914450    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="331148a9-a72e-48fa-8609-9bbb348ee04a" path="/var/lib/kubelet/pods/331148a9-a72e-48fa-8609-9bbb348ee04a/volumes"
Sep 17 03:27:52.195231 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.195180    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-584c46bf5b-6cbgq"]
Sep 17 03:27:52.195581 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.195549    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="331148a9-a72e-48fa-8609-9bbb348ee04a" containerName="authorino"
Sep 17 03:27:52.195581 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.195560    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="331148a9-a72e-48fa-8609-9bbb348ee04a" containerName="authorino"
Sep 17 03:27:52.195651 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.195614    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="331148a9-a72e-48fa-8609-9bbb348ee04a" containerName="authorino"
Sep 17 03:27:52.219017 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.218993    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-584c46bf5b-6cbgq"]
Sep 17 03:27:52.219159 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.219077    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:27:52.221376 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.221338    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 17 03:27:52.222538 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.222064    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 17 03:27:52.222538 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.222341    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j9jwv\""
Sep 17 03:27:52.222538 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.222406    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 17 03:27:52.336952 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.336918    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sc87\" (UniqueName: \"kubernetes.io/projected/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-kube-api-access-7sc87\") pod \"maas-controller-584c46bf5b-6cbgq\" (UID: \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\") " pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:27:52.337074 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.336998    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-webhook-cert\") pod \"maas-controller-584c46bf5b-6cbgq\" (UID: \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\") " pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:27:52.337074 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.337040    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-metrics-certs\") pod \"maas-controller-584c46bf5b-6cbgq\" (UID: \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\") " pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:27:52.374808 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.374781    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c8c8d6b99-xc8jh"]
Sep 17 03:27:52.378248 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.378227    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh"
Sep 17 03:27:52.386635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.386608    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c8c8d6b99-xc8jh"]
Sep 17 03:27:52.438302 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.438282    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sc87\" (UniqueName: \"kubernetes.io/projected/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-kube-api-access-7sc87\") pod \"maas-controller-584c46bf5b-6cbgq\" (UID: \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\") " pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:27:52.438399 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.438314    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkbgz\" (UniqueName: \"kubernetes.io/projected/b9c51d77-1a0c-41df-aa7a-43c46772e496-kube-api-access-pkbgz\") pod \"maas-controller-7c8c8d6b99-xc8jh\" (UID: \"b9c51d77-1a0c-41df-aa7a-43c46772e496\") " pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh"
Sep 17 03:27:52.438399 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.438332    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c51d77-1a0c-41df-aa7a-43c46772e496-webhook-cert\") pod \"maas-controller-7c8c8d6b99-xc8jh\" (UID: \"b9c51d77-1a0c-41df-aa7a-43c46772e496\") " pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh"
Sep 17 03:27:52.438475 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.438445    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c51d77-1a0c-41df-aa7a-43c46772e496-metrics-certs\") pod \"maas-controller-7c8c8d6b99-xc8jh\" (UID: \"b9c51d77-1a0c-41df-aa7a-43c46772e496\") " pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh"
Sep 17 03:27:52.438544 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.438529    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-webhook-cert\") pod \"maas-controller-584c46bf5b-6cbgq\" (UID: \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\") " pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:27:52.438588 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.438569    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-metrics-certs\") pod \"maas-controller-584c46bf5b-6cbgq\" (UID: \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\") " pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:27:52.438713 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:27:52.438696    2557 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 17 03:27:52.438783 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:27:52.438761    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-metrics-certs podName:0ea440ff-5d7d-4e02-8d34-8039e8a546a9 nodeName:}" failed. No retries permitted until 2026-09-17 03:27:52.938741068 +0000 UTC m=+523.541837057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-metrics-certs") pod "maas-controller-584c46bf5b-6cbgq" (UID: "0ea440ff-5d7d-4e02-8d34-8039e8a546a9") : secret "maas-controller-metrics-tls" not found
Sep 17 03:27:52.441082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.441061    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-webhook-cert\") pod \"maas-controller-584c46bf5b-6cbgq\" (UID: \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\") " pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:27:52.446844 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.446799    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sc87\" (UniqueName: \"kubernetes.io/projected/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-kube-api-access-7sc87\") pod \"maas-controller-584c46bf5b-6cbgq\" (UID: \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\") " pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:27:52.539381 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.539347    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkbgz\" (UniqueName: \"kubernetes.io/projected/b9c51d77-1a0c-41df-aa7a-43c46772e496-kube-api-access-pkbgz\") pod \"maas-controller-7c8c8d6b99-xc8jh\" (UID: \"b9c51d77-1a0c-41df-aa7a-43c46772e496\") " pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh"
Sep 17 03:27:52.539381 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.539385    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c51d77-1a0c-41df-aa7a-43c46772e496-webhook-cert\") pod \"maas-controller-7c8c8d6b99-xc8jh\" (UID: \"b9c51d77-1a0c-41df-aa7a-43c46772e496\") " pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh"
Sep 17 03:27:52.539523 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.539413    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c51d77-1a0c-41df-aa7a-43c46772e496-metrics-certs\") pod \"maas-controller-7c8c8d6b99-xc8jh\" (UID: \"b9c51d77-1a0c-41df-aa7a-43c46772e496\") " pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh"
Sep 17 03:27:52.541552 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.541535    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c51d77-1a0c-41df-aa7a-43c46772e496-webhook-cert\") pod \"maas-controller-7c8c8d6b99-xc8jh\" (UID: \"b9c51d77-1a0c-41df-aa7a-43c46772e496\") " pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh"
Sep 17 03:27:52.541706 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.541687    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c51d77-1a0c-41df-aa7a-43c46772e496-metrics-certs\") pod \"maas-controller-7c8c8d6b99-xc8jh\" (UID: \"b9c51d77-1a0c-41df-aa7a-43c46772e496\") " pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh"
Sep 17 03:27:52.547068 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.547048    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkbgz\" (UniqueName: \"kubernetes.io/projected/b9c51d77-1a0c-41df-aa7a-43c46772e496-kube-api-access-pkbgz\") pod \"maas-controller-7c8c8d6b99-xc8jh\" (UID: \"b9c51d77-1a0c-41df-aa7a-43c46772e496\") " pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh"
Sep 17 03:27:52.689405 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.689385    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh"
Sep 17 03:27:52.943393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.943366    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-metrics-certs\") pod \"maas-controller-584c46bf5b-6cbgq\" (UID: \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\") " pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:27:52.945539 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.945521    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-metrics-certs\") pod \"maas-controller-584c46bf5b-6cbgq\" (UID: \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\") " pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:27:52.960644 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:52.960625    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c8c8d6b99-xc8jh"]
Sep 17 03:27:52.962538 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:27:52.962516    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c51d77_1a0c_41df_aa7a_43c46772e496.slice/crio-bbecab56f70d6182554928c5c60eb3977f3c775f21e4261577021ecc01918a3e WatchSource:0}: Error finding container bbecab56f70d6182554928c5c60eb3977f3c775f21e4261577021ecc01918a3e: Status 404 returned error can't find the container with id bbecab56f70d6182554928c5c60eb3977f3c775f21e4261577021ecc01918a3e
Sep 17 03:27:53.130185 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:53.130161    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:27:53.357626 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:53.357558    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh" event={"ID":"b9c51d77-1a0c-41df-aa7a-43c46772e496","Type":"ContainerStarted","Data":"bbecab56f70d6182554928c5c60eb3977f3c775f21e4261577021ecc01918a3e"}
Sep 17 03:27:53.400884 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:53.400862    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-584c46bf5b-6cbgq"]
Sep 17 03:27:53.402628 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:27:53.402606    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea440ff_5d7d_4e02_8d34_8039e8a546a9.slice/crio-bae4f2c036ca0b16f86a364c2bdc46d175e4458a7145acb99cf4274337de1e84 WatchSource:0}: Error finding container bae4f2c036ca0b16f86a364c2bdc46d175e4458a7145acb99cf4274337de1e84: Status 404 returned error can't find the container with id bae4f2c036ca0b16f86a364c2bdc46d175e4458a7145acb99cf4274337de1e84
Sep 17 03:27:54.367183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:54.367129    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-584c46bf5b-6cbgq" event={"ID":"0ea440ff-5d7d-4e02-8d34-8039e8a546a9","Type":"ContainerStarted","Data":"bae4f2c036ca0b16f86a364c2bdc46d175e4458a7145acb99cf4274337de1e84"}
Sep 17 03:27:56.379310 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:56.379266    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh" event={"ID":"b9c51d77-1a0c-41df-aa7a-43c46772e496","Type":"ContainerStarted","Data":"b64d21b3110e6e66fea9dacb8f14d429f9b4a54848935b967fe6748ffe5911ac"}
Sep 17 03:27:56.379767 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:56.379429    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh"
Sep 17 03:27:56.380896 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:56.380873    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-584c46bf5b-6cbgq" event={"ID":"0ea440ff-5d7d-4e02-8d34-8039e8a546a9","Type":"ContainerStarted","Data":"e9081a58bb5c6238a22eeee0b22d58f9a632092e946f48faab5725f8f197bb4d"}
Sep 17 03:27:56.381010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:56.380912    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:27:56.419905 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:56.419860    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh" podStartSLOduration=1.731630759 podStartE2EDuration="4.419845957s" totalImagesPullingTime="2.688215198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:27:56.418175818 +0000 UTC m=+527.021271821" watchObservedRunningTime="2026-09-17 03:27:56.419845957 +0000 UTC m=+527.022941961"
Sep 17 03:27:56.440710 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:27:56.440670    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-584c46bf5b-6cbgq" podStartSLOduration=2.182064689 podStartE2EDuration="4.440658831s" totalImagesPullingTime="2.258594142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:27:56.438715145 +0000 UTC m=+527.041811149" watchObservedRunningTime="2026-09-17 03:27:56.440658831 +0000 UTC m=+527.043754835"
Sep 17 03:28:07.387405 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:07.387303    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh"
Sep 17 03:28:07.387868 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:07.387515    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:28:07.447762 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:07.447731    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-584c46bf5b-6cbgq"]
Sep 17 03:28:07.447956 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:07.447909    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-584c46bf5b-6cbgq" podUID="0ea440ff-5d7d-4e02-8d34-8039e8a546a9" containerName="manager" containerID="cri-o://e9081a58bb5c6238a22eeee0b22d58f9a632092e946f48faab5725f8f197bb4d" gracePeriod=30
Sep 17 03:28:07.687559 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:07.687532    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:28:07.767115 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:07.767072    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sc87\" (UniqueName: \"kubernetes.io/projected/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-kube-api-access-7sc87\") pod \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\" (UID: \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\") "
Sep 17 03:28:07.767276 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:07.767123    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-metrics-certs\") pod \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\" (UID: \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\") "
Sep 17 03:28:07.767276 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:07.767255    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-webhook-cert\") pod \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\" (UID: \"0ea440ff-5d7d-4e02-8d34-8039e8a546a9\") "
Sep 17 03:28:07.769278 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:07.769251    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0ea440ff-5d7d-4e02-8d34-8039e8a546a9" (UID: "0ea440ff-5d7d-4e02-8d34-8039e8a546a9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:28:07.769401 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:07.769322    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0ea440ff-5d7d-4e02-8d34-8039e8a546a9" (UID: "0ea440ff-5d7d-4e02-8d34-8039e8a546a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:28:07.769401 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:07.769351    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-kube-api-access-7sc87" (OuterVolumeSpecName: "kube-api-access-7sc87") pod "0ea440ff-5d7d-4e02-8d34-8039e8a546a9" (UID: "0ea440ff-5d7d-4e02-8d34-8039e8a546a9"). InnerVolumeSpecName "kube-api-access-7sc87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:28:07.867794 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:07.867767    2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-webhook-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:28:07.867794 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:07.867793    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sc87\" (UniqueName: \"kubernetes.io/projected/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-kube-api-access-7sc87\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:28:07.867931 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:07.867808    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ea440ff-5d7d-4e02-8d34-8039e8a546a9-metrics-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:28:08.439006 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:08.438970    2557 generic.go:356] "Generic (PLEG): container finished" podID="0ea440ff-5d7d-4e02-8d34-8039e8a546a9" containerID="e9081a58bb5c6238a22eeee0b22d58f9a632092e946f48faab5725f8f197bb4d" exitCode=0
Sep 17 03:28:08.439386 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:08.439017    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-584c46bf5b-6cbgq"
Sep 17 03:28:08.439386 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:08.439064    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-584c46bf5b-6cbgq" event={"ID":"0ea440ff-5d7d-4e02-8d34-8039e8a546a9","Type":"ContainerDied","Data":"e9081a58bb5c6238a22eeee0b22d58f9a632092e946f48faab5725f8f197bb4d"}
Sep 17 03:28:08.439386 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:08.439128    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-584c46bf5b-6cbgq" event={"ID":"0ea440ff-5d7d-4e02-8d34-8039e8a546a9","Type":"ContainerDied","Data":"bae4f2c036ca0b16f86a364c2bdc46d175e4458a7145acb99cf4274337de1e84"}
Sep 17 03:28:08.439386 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:08.439155    2557 scope.go:117] "RemoveContainer" containerID="e9081a58bb5c6238a22eeee0b22d58f9a632092e946f48faab5725f8f197bb4d"
Sep 17 03:28:08.448867 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:08.448645    2557 scope.go:117] "RemoveContainer" containerID="e9081a58bb5c6238a22eeee0b22d58f9a632092e946f48faab5725f8f197bb4d"
Sep 17 03:28:08.449022 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:28:08.449002    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9081a58bb5c6238a22eeee0b22d58f9a632092e946f48faab5725f8f197bb4d\": container with ID starting with e9081a58bb5c6238a22eeee0b22d58f9a632092e946f48faab5725f8f197bb4d not found: ID does not exist" containerID="e9081a58bb5c6238a22eeee0b22d58f9a632092e946f48faab5725f8f197bb4d"
Sep 17 03:28:08.449070 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:08.449028    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9081a58bb5c6238a22eeee0b22d58f9a632092e946f48faab5725f8f197bb4d"} err="failed to get container status \"e9081a58bb5c6238a22eeee0b22d58f9a632092e946f48faab5725f8f197bb4d\": rpc error: code = NotFound desc = could not find container \"e9081a58bb5c6238a22eeee0b22d58f9a632092e946f48faab5725f8f197bb4d\": container with ID starting with e9081a58bb5c6238a22eeee0b22d58f9a632092e946f48faab5725f8f197bb4d not found: ID does not exist"
Sep 17 03:28:08.453766 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:08.453739    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-584c46bf5b-6cbgq"]
Sep 17 03:28:08.457654 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:08.457631    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-584c46bf5b-6cbgq"]
Sep 17 03:28:09.915968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:09.915942    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ea440ff-5d7d-4e02-8d34-8039e8a546a9" path="/var/lib/kubelet/pods/0ea440ff-5d7d-4e02-8d34-8039e8a546a9/volumes"
Sep 17 03:28:32.146928 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.146892    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"]
Sep 17 03:28:32.147413 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.147274    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ea440ff-5d7d-4e02-8d34-8039e8a546a9" containerName="manager"
Sep 17 03:28:32.147413 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.147288    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ea440ff-5d7d-4e02-8d34-8039e8a546a9" containerName="manager"
Sep 17 03:28:32.147902 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.147871    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ea440ff-5d7d-4e02-8d34-8039e8a546a9" containerName="manager"
Sep 17 03:28:32.156636 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.156592    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:32.159479 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.159453    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 17 03:28:32.159625 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.159483    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"]
Sep 17 03:28:32.159625 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.159511    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 17 03:28:32.159625 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.159453    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-trmz9\""
Sep 17 03:28:32.266812 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.266782    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b28718-2cb2-473b-add7-d5e23b45b900-metrics-certs\") pod \"maas-api-5575f6cfb4-c6jvh\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:32.266937 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.266820    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mkvv\" (UniqueName: \"kubernetes.io/projected/47b28718-2cb2-473b-add7-d5e23b45b900-kube-api-access-7mkvv\") pod \"maas-api-5575f6cfb4-c6jvh\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:32.266937 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.266916    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b28718-2cb2-473b-add7-d5e23b45b900-maas-api-tls\") pod \"maas-api-5575f6cfb4-c6jvh\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:32.267016 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.266972    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b28718-2cb2-473b-add7-d5e23b45b900-openshift-service-ca\") pod \"maas-api-5575f6cfb4-c6jvh\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:32.367416 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.367392    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b28718-2cb2-473b-add7-d5e23b45b900-maas-api-tls\") pod \"maas-api-5575f6cfb4-c6jvh\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:32.367536 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.367438    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b28718-2cb2-473b-add7-d5e23b45b900-openshift-service-ca\") pod \"maas-api-5575f6cfb4-c6jvh\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:32.367536 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.367458    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b28718-2cb2-473b-add7-d5e23b45b900-metrics-certs\") pod \"maas-api-5575f6cfb4-c6jvh\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:32.367536 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.367478    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mkvv\" (UniqueName: \"kubernetes.io/projected/47b28718-2cb2-473b-add7-d5e23b45b900-kube-api-access-7mkvv\") pod \"maas-api-5575f6cfb4-c6jvh\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:32.367997 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.367978    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b28718-2cb2-473b-add7-d5e23b45b900-openshift-service-ca\") pod \"maas-api-5575f6cfb4-c6jvh\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:32.369854 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.369835    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b28718-2cb2-473b-add7-d5e23b45b900-metrics-certs\") pod \"maas-api-5575f6cfb4-c6jvh\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:32.369930 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.369866    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b28718-2cb2-473b-add7-d5e23b45b900-maas-api-tls\") pod \"maas-api-5575f6cfb4-c6jvh\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:32.380535 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.380508    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mkvv\" (UniqueName: \"kubernetes.io/projected/47b28718-2cb2-473b-add7-d5e23b45b900-kube-api-access-7mkvv\") pod \"maas-api-5575f6cfb4-c6jvh\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") " pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:32.472163 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.472107    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:32.672994 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.672961    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62"]
Sep 17 03:28:32.676618 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.676600    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62"
Sep 17 03:28:32.678530 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.678508    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 17 03:28:32.686421 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.686400    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62"]
Sep 17 03:28:32.698253 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.698228    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7gscq"]
Sep 17 03:28:32.701639 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.701620    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7gscq"
Sep 17 03:28:32.703942 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.703925    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2zpzv\""
Sep 17 03:28:32.710968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.710947    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7gscq"]
Sep 17 03:28:32.744083 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.744065    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"]
Sep 17 03:28:32.746326 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:28:32.746304    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b28718_2cb2_473b_add7_d5e23b45b900.slice/crio-6c80537a65143d786cb860b7d244bbb2d4ef4413e4a0a2112b6ce9a663c925cb WatchSource:0}: Error finding container 6c80537a65143d786cb860b7d244bbb2d4ef4413e4a0a2112b6ce9a663c925cb: Status 404 returned error can't find the container with id 6c80537a65143d786cb860b7d244bbb2d4ef4413e4a0a2112b6ce9a663c925cb
Sep 17 03:28:32.770631 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.770612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwqw7\" (UniqueName: \"kubernetes.io/projected/1dd3bf10-7094-4cb1-8a06-7dfbb62131ae-kube-api-access-kwqw7\") pod \"payload-pre-processing-59fbc47bf8-j9c62\" (UID: \"1dd3bf10-7094-4cb1-8a06-7dfbb62131ae\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62"
Sep 17 03:28:32.770715 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.770655    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd3bf10-7094-4cb1-8a06-7dfbb62131ae-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-j9c62\" (UID: \"1dd3bf10-7094-4cb1-8a06-7dfbb62131ae\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62"
Sep 17 03:28:32.871750 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.871716    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd3bf10-7094-4cb1-8a06-7dfbb62131ae-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-j9c62\" (UID: \"1dd3bf10-7094-4cb1-8a06-7dfbb62131ae\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62"
Sep 17 03:28:32.871902 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.871816    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7gscq\" (UID: \"6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7gscq"
Sep 17 03:28:32.871902 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.871854    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwqw7\" (UniqueName: \"kubernetes.io/projected/1dd3bf10-7094-4cb1-8a06-7dfbb62131ae-kube-api-access-kwqw7\") pod \"payload-pre-processing-59fbc47bf8-j9c62\" (UID: \"1dd3bf10-7094-4cb1-8a06-7dfbb62131ae\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62"
Sep 17 03:28:32.871902 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.871879    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhn2d\" (UniqueName: \"kubernetes.io/projected/6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb-kube-api-access-rhn2d\") pod \"payload-processing-5b54b7fbb7-7gscq\" (UID: \"6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7gscq"
Sep 17 03:28:32.872435 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.872416    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd3bf10-7094-4cb1-8a06-7dfbb62131ae-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-j9c62\" (UID: \"1dd3bf10-7094-4cb1-8a06-7dfbb62131ae\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62"
Sep 17 03:28:32.880069 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.880050    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwqw7\" (UniqueName: \"kubernetes.io/projected/1dd3bf10-7094-4cb1-8a06-7dfbb62131ae-kube-api-access-kwqw7\") pod \"payload-pre-processing-59fbc47bf8-j9c62\" (UID: \"1dd3bf10-7094-4cb1-8a06-7dfbb62131ae\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62"
Sep 17 03:28:32.973326 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.973290    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7gscq\" (UID: \"6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7gscq"
Sep 17 03:28:32.973464 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.973337    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhn2d\" (UniqueName: \"kubernetes.io/projected/6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb-kube-api-access-rhn2d\") pod \"payload-processing-5b54b7fbb7-7gscq\" (UID: \"6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7gscq"
Sep 17 03:28:32.974152 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.974132    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7gscq\" (UID: \"6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7gscq"
Sep 17 03:28:32.983536 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.983517    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhn2d\" (UniqueName: \"kubernetes.io/projected/6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb-kube-api-access-rhn2d\") pod \"payload-processing-5b54b7fbb7-7gscq\" (UID: \"6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7gscq"
Sep 17 03:28:32.988328 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:32.988301    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62"
Sep 17 03:28:33.012054 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:33.012013    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7gscq"
Sep 17 03:28:33.260657 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:33.260633    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62"]
Sep 17 03:28:33.262737 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:28:33.262713    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd3bf10_7094_4cb1_8a06_7dfbb62131ae.slice/crio-2c8480d1a14e7adb378a8f010f5bbe44003b9f97b78566b1922f146a59c792e7 WatchSource:0}: Error finding container 2c8480d1a14e7adb378a8f010f5bbe44003b9f97b78566b1922f146a59c792e7: Status 404 returned error can't find the container with id 2c8480d1a14e7adb378a8f010f5bbe44003b9f97b78566b1922f146a59c792e7
Sep 17 03:28:33.284850 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:33.284824    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7gscq"]
Sep 17 03:28:33.288471 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:28:33.288445    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aacdd63_3c26_4e1c_bd26_0a5a7cffcdcb.slice/crio-3efe1556e744ab66ee81cf620f3bdad7607237bed4644b4bab9730f159ecf6f2 WatchSource:0}: Error finding container 3efe1556e744ab66ee81cf620f3bdad7607237bed4644b4bab9730f159ecf6f2: Status 404 returned error can't find the container with id 3efe1556e744ab66ee81cf620f3bdad7607237bed4644b4bab9730f159ecf6f2
Sep 17 03:28:33.573379 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:33.573296    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh" event={"ID":"47b28718-2cb2-473b-add7-d5e23b45b900","Type":"ContainerStarted","Data":"6c80537a65143d786cb860b7d244bbb2d4ef4413e4a0a2112b6ce9a663c925cb"}
Sep 17 03:28:33.574847 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:33.574817    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7gscq" event={"ID":"6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb","Type":"ContainerStarted","Data":"3efe1556e744ab66ee81cf620f3bdad7607237bed4644b4bab9730f159ecf6f2"}
Sep 17 03:28:33.576258 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:33.576230    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62" event={"ID":"1dd3bf10-7094-4cb1-8a06-7dfbb62131ae","Type":"ContainerStarted","Data":"2c8480d1a14e7adb378a8f010f5bbe44003b9f97b78566b1922f146a59c792e7"}
Sep 17 03:28:35.589335 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:35.589232    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh" event={"ID":"47b28718-2cb2-473b-add7-d5e23b45b900","Type":"ContainerStarted","Data":"530aeaae4410c67750f5221a35a95c81c27600588a8eb654a1b212348edaa992"}
Sep 17 03:28:35.589335 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:35.589316    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:35.590920 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:35.590890    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7gscq" event={"ID":"6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb","Type":"ContainerStarted","Data":"95a3a0e1b3344618c5c585eb97d5e19424d1fcd951df1c2d0c5f2a3ebc29b77b"}
Sep 17 03:28:35.591038 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:35.590934    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7gscq"
Sep 17 03:28:35.592576 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:35.592550    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62" event={"ID":"1dd3bf10-7094-4cb1-8a06-7dfbb62131ae","Type":"ContainerStarted","Data":"061cfd19fecfc21bac0fac5a27d339fc54c6f65bd6a21a5199f27313854ca55d"}
Sep 17 03:28:35.592685 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:35.592604    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62"
Sep 17 03:28:35.608599 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:35.608552    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh" podStartSLOduration=1.161422249 podStartE2EDuration="3.60853417s" totalImagesPullingTime="2.447111921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:28:35.605645587 +0000 UTC m=+566.208741619" watchObservedRunningTime="2026-09-17 03:28:35.60853417 +0000 UTC m=+566.211630178"
Sep 17 03:28:35.624692 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:35.624638    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-7gscq" podStartSLOduration=1.720132593 podStartE2EDuration="3.62459511s" totalImagesPullingTime="1.904462517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:28:35.622389839 +0000 UTC m=+566.225485837" watchObservedRunningTime="2026-09-17 03:28:35.62459511 +0000 UTC m=+566.227691118"
Sep 17 03:28:41.603024 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:41.602996    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:28:41.620124 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:41.620077    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62" podStartSLOduration=7.687748641 podStartE2EDuration="9.620062094s" totalImagesPullingTime="1.932313453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:28:35.640673217 +0000 UTC m=+566.243769222" watchObservedRunningTime="2026-09-17 03:28:41.620062094 +0000 UTC m=+572.223158098"
Sep 17 03:28:42.628876 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:42.628850    2557 generic.go:356] "Generic (PLEG): container finished" podID="f1a4d22e-eade-4378-bdc4-5f42eb080071" containerID="68e83da339405be096476a6bc528ba6d43ad79be7e7680af2e2e9392a8a23d88" exitCode=0
Sep 17 03:28:42.629274 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:42.628922    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx" event={"ID":"f1a4d22e-eade-4378-bdc4-5f42eb080071","Type":"ContainerDied","Data":"68e83da339405be096476a6bc528ba6d43ad79be7e7680af2e2e9392a8a23d88"}
Sep 17 03:28:42.629372 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:42.629352    2557 scope.go:117] "RemoveContainer" containerID="68e83da339405be096476a6bc528ba6d43ad79be7e7680af2e2e9392a8a23d88"
Sep 17 03:28:42.630317 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:42.630289    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:28:42.630386 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:42.630358    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:28:42.630434 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:42.630389    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:28:43.640616 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:43.639666    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx" event={"ID":"f1a4d22e-eade-4378-bdc4-5f42eb080071","Type":"ContainerStarted","Data":"411b49609bf20a99c737870d8afb0865d5ac55c9844abcecaaae38a74692b7c5"}
Sep 17 03:28:43.746623 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:43.746571    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:28:43.746904 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:43.746878    2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-q9jxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 17 03:28:43.746968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:43.746937    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx" podUID="f1a4d22e-eade-4378-bdc4-5f42eb080071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 17 03:28:44.746434 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:44.746382    2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-q9jxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 17 03:28:44.746936 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:44.746458    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx" podUID="f1a4d22e-eade-4378-bdc4-5f42eb080071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 17 03:28:45.746362 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:45.746329    2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-q9jxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 17 03:28:45.746552 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:45.746385    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx" podUID="f1a4d22e-eade-4378-bdc4-5f42eb080071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 17 03:28:46.599729 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:46.599695    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j9c62"
Sep 17 03:28:46.599984 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:46.599963    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7gscq"
Sep 17 03:28:46.746646 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:46.746610    2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-q9jxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 17 03:28:46.747101 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:46.746671    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx" podUID="f1a4d22e-eade-4378-bdc4-5f42eb080071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 17 03:28:47.746252 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:47.746218    2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-q9jxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 17 03:28:47.746430 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:47.746294    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx" podUID="f1a4d22e-eade-4378-bdc4-5f42eb080071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 17 03:28:48.749985 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:48.749956    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:28:48.750378 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:48.750225    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:28:48.750966 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:48.750948    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-q9jxx"
Sep 17 03:28:52.293534 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.293499    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"]
Sep 17 03:28:52.298049 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.298030    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.301214 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.301169    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 03:28:52.301214 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.301174    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 03:28:52.301400 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.301224    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-g74zs\""
Sep 17 03:28:52.301400 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.301251    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 17 03:28:52.305880 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.305752    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"]
Sep 17 03:28:52.337509 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.337487    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9378a674-d679-4414-ba82-0f364ba4a164-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.337615 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.337515    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9378a674-d679-4414-ba82-0f364ba4a164-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.337615 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.337572    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9378a674-d679-4414-ba82-0f364ba4a164-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.337710 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.337615    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9378a674-d679-4414-ba82-0f364ba4a164-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.337710 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.337669    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9378a674-d679-4414-ba82-0f364ba4a164-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.337795 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.337709    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftv7l\" (UniqueName: \"kubernetes.io/projected/9378a674-d679-4414-ba82-0f364ba4a164-kube-api-access-ftv7l\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.438477 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.438449    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftv7l\" (UniqueName: \"kubernetes.io/projected/9378a674-d679-4414-ba82-0f364ba4a164-kube-api-access-ftv7l\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.438587 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.438502    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9378a674-d679-4414-ba82-0f364ba4a164-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.438587 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.438528    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9378a674-d679-4414-ba82-0f364ba4a164-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.438675 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.438625    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9378a674-d679-4414-ba82-0f364ba4a164-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.438675 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.438662    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9378a674-d679-4414-ba82-0f364ba4a164-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.438752 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.438701    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9378a674-d679-4414-ba82-0f364ba4a164-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.438953 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.438931    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9378a674-d679-4414-ba82-0f364ba4a164-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.439022 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.438979    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9378a674-d679-4414-ba82-0f364ba4a164-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.439136 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.439075    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9378a674-d679-4414-ba82-0f364ba4a164-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.440890 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.440865    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9378a674-d679-4414-ba82-0f364ba4a164-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.441132 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.441115    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9378a674-d679-4414-ba82-0f364ba4a164-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.452217 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.452175    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftv7l\" (UniqueName: \"kubernetes.io/projected/9378a674-d679-4414-ba82-0f364ba4a164-kube-api-access-ftv7l\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9\" (UID: \"9378a674-d679-4414-ba82-0f364ba4a164\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.609845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.609784    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:28:52.880219 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:52.880179    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"]
Sep 17 03:28:52.882292 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:28:52.882257    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9378a674_d679_4414_ba82_0f364ba4a164.slice/crio-0794e4303674785f5a1db615b4c259d6c3fcd2470b1df79c1ef8c73d58a8ec6d WatchSource:0}: Error finding container 0794e4303674785f5a1db615b4c259d6c3fcd2470b1df79c1ef8c73d58a8ec6d: Status 404 returned error can't find the container with id 0794e4303674785f5a1db615b4c259d6c3fcd2470b1df79c1ef8c73d58a8ec6d
Sep 17 03:28:53.700960 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:53.700926    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9" event={"ID":"9378a674-d679-4414-ba82-0f364ba4a164","Type":"ContainerStarted","Data":"0794e4303674785f5a1db615b4c259d6c3fcd2470b1df79c1ef8c73d58a8ec6d"}
Sep 17 03:28:54.707630 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:54.707594    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9" event={"ID":"9378a674-d679-4414-ba82-0f364ba4a164","Type":"ContainerStarted","Data":"e163e8610539be0fa6955d3cc1f8de794579d5bfbad95f5ed03a16e3dcc99748"}
Sep 17 03:28:54.747744 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:28:54.747700    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9" podStartSLOduration=1.762726936 podStartE2EDuration="2.747684069s" totalImagesPullingTime="984.957133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:28:54.746016388 +0000 UTC m=+585.349112393" watchObservedRunningTime="2026-09-17 03:28:54.747684069 +0000 UTC m=+585.350780074"
Sep 17 03:29:02.610247 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:02.610186    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:29:02.610806 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:02.610273    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:29:02.622948 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:02.622923    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:29:02.757482 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:02.757453    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9"
Sep 17 03:29:09.853405 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:09.853374    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:29:09.854881 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:09.854859    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:29:22.678024 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.677994    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"]
Sep 17 03:29:22.682701 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.682681    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.684880 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.684860    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 17 03:29:22.691486 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.691463    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"]
Sep 17 03:29:22.797007 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.796984    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.797142 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.797035    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.797142 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.797056    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qndd9\" (UniqueName: \"kubernetes.io/projected/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-kube-api-access-qndd9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.797142 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.797130    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.797268 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.797160    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.797268 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.797177    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.898448 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.898412    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.898613 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.898454    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndd9\" (UniqueName: \"kubernetes.io/projected/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-kube-api-access-qndd9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.898613 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.898479    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.898752 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.898727    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.898816 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.898772    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.898879 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.898866    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.899041 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.899019    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.899131 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.899110    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.899227 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.899186    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.900644 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.900623    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.901100 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.901083    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.907232 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.907211    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qndd9\" (UniqueName: \"kubernetes.io/projected/ce023e03-a8b3-4c4d-b46c-5dd04ef2434f-kube-api-access-qndd9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-glxxj\" (UID: \"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:22.994231 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:22.994152    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:23.266332 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:23.266307    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"]
Sep 17 03:29:23.268672 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:29:23.268643    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce023e03_a8b3_4c4d_b46c_5dd04ef2434f.slice/crio-16d0c07ccd23e595cd030d1e5c5c4284ff00efae725fc938c457aa945a5efa06 WatchSource:0}: Error finding container 16d0c07ccd23e595cd030d1e5c5c4284ff00efae725fc938c457aa945a5efa06: Status 404 returned error can't find the container with id 16d0c07ccd23e595cd030d1e5c5c4284ff00efae725fc938c457aa945a5efa06
Sep 17 03:29:23.270297 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:23.270276    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 03:29:23.848853 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:23.848820    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj" event={"ID":"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f","Type":"ContainerStarted","Data":"6cc5b7bda23baf2ee5efff17469212fe79aa3eb3ab98e15704d4c91df173b49e"}
Sep 17 03:29:23.849318 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:23.848874    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj" event={"ID":"ce023e03-a8b3-4c4d-b46c-5dd04ef2434f","Type":"ContainerStarted","Data":"16d0c07ccd23e595cd030d1e5c5c4284ff00efae725fc938c457aa945a5efa06"}
Sep 17 03:29:23.867075 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:23.867031    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj" podStartSLOduration=1.69684338 podStartE2EDuration="1.867011851s" totalImagesPullingTime="170.168471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:29:23.866682865 +0000 UTC m=+614.469778870" watchObservedRunningTime="2026-09-17 03:29:23.867011851 +0000 UTC m=+614.470107856"
Sep 17 03:29:32.994319 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:32.994234    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:32.994319 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:32.994286    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:33.006273 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:33.006249    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:33.914403 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:33.914380    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-glxxj"
Sep 17 03:29:34.289511 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.289434    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"]
Sep 17 03:29:34.294590 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.294566    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.296727 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.296706    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 17 03:29:34.301448 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.301424    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"]
Sep 17 03:29:34.389630 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.389606    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.389753 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.389674    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdrjq\" (UniqueName: \"kubernetes.io/projected/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-kube-api-access-fdrjq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.389753 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.389712    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.389753 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.389732    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.389911 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.389761    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.389911 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.389876    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.490482 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.490454    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.490602 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.490488    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.490602 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.490509    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.490786 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.490764    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.490846 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.490830    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.490924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.490904    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.490971 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.490936    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.490971 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.490954    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdrjq\" (UniqueName: \"kubernetes.io/projected/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-kube-api-access-fdrjq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.491074 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.490975    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.493036 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.493014    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.493179 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.493164    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.498550 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.498530    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdrjq\" (UniqueName: \"kubernetes.io/projected/ea85fde4-8a1c-4e42-9a32-1d799a3f0041-kube-api-access-fdrjq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc\" (UID: \"ea85fde4-8a1c-4e42-9a32-1d799a3f0041\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.605811 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.605755    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:34.874923 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.874902    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"]
Sep 17 03:29:34.876971 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:29:34.876945    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea85fde4_8a1c_4e42_9a32_1d799a3f0041.slice/crio-fa012a3218591bdd6f6b0e7926bc5e2bf922bdeae173eaf012b530cf2b1e65f9 WatchSource:0}: Error finding container fa012a3218591bdd6f6b0e7926bc5e2bf922bdeae173eaf012b530cf2b1e65f9: Status 404 returned error can't find the container with id fa012a3218591bdd6f6b0e7926bc5e2bf922bdeae173eaf012b530cf2b1e65f9
Sep 17 03:29:34.903841 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:34.903816    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc" event={"ID":"ea85fde4-8a1c-4e42-9a32-1d799a3f0041","Type":"ContainerStarted","Data":"fa012a3218591bdd6f6b0e7926bc5e2bf922bdeae173eaf012b530cf2b1e65f9"}
Sep 17 03:29:35.914952 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:35.914920    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc" event={"ID":"ea85fde4-8a1c-4e42-9a32-1d799a3f0041","Type":"ContainerStarted","Data":"86eb098757ee5c4a15e72b10eb1a3fc640239daec47806808173d2dd84262741"}
Sep 17 03:29:35.931855 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:35.931809    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc" podStartSLOduration=1.762380758 podStartE2EDuration="1.931794124s" totalImagesPullingTime="169.413366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:29:35.929972964 +0000 UTC m=+626.533068969" watchObservedRunningTime="2026-09-17 03:29:35.931794124 +0000 UTC m=+626.534890125"
Sep 17 03:29:44.606126 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:44.606088    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:44.606594 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:44.606152    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:44.618624 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:44.618600    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:29:44.969562 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:29:44.969536    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc"
Sep 17 03:31:41.609996 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.609961    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"]
Sep 17 03:31:41.613770 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.613741    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.615801 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.615779    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-214fea-openshift-default-dockercfg-6lxxp\""
Sep 17 03:31:41.628303 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.628284    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"]
Sep 17 03:31:41.684482 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.684450    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"]
Sep 17 03:31:41.688267 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.688244    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.691549 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.691514    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1afd34-openshift-default-dockercfg-2dq8c\""
Sep 17 03:31:41.716075 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.716052    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"]
Sep 17 03:31:41.722461 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.722435    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istiod-ca-cert\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.722598 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.722570    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-envoy\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.722690 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.722612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-podinfo\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.722690 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.722673    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-credential-socket\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.722804 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.722717    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-workload-socket\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.722804 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.722742    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-workload-certs\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.722804 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.722783    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-data\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.722958 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.722807    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-token\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.722958 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.722836    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx2lh\" (UniqueName: \"kubernetes.io/projected/740bc52a-9b17-4fe7-ba69-fe88ab56706d-kube-api-access-jx2lh\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.823214 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.823152    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"]
Sep 17 03:31:41.824470 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.824426    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-envoy\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.825146 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825116    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-envoy\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.825307 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825180    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-podinfo\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.825307 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825269    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-credential-socket\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.825480 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825332    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-token\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.825480 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825401    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-credential-socket\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.825480 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825445    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-workload-socket\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.825480 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825469    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-workload-certs\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.825480 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825494    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-data\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.827446 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825530    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-workload-socket\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.827446 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825566    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-data\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.827446 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825589    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-token\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.827446 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-podinfo\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.827446 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825638    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx2lh\" (UniqueName: \"kubernetes.io/projected/740bc52a-9b17-4fe7-ba69-fe88ab56706d-kube-api-access-jx2lh\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.827446 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825668    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-workload-certs\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.827446 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825708    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm9z9\" (UniqueName: \"kubernetes.io/projected/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-kube-api-access-dm9z9\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.827446 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825773    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istiod-ca-cert\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.827446 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.825818    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istiod-ca-cert\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.827446 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.826219    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-credential-socket\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.827446 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.826482    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-workload-socket\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.827446 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.826738    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-workload-certs\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.827446 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.826989    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-data\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.829801 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.827649    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istiod-ca-cert\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.829801 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.828023    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:41.830065 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.830046    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-podinfo\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.830339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.830322    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-envoy\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.831899 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.831882    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-297324-openshift-default-dockercfg-bz7tf\""
Sep 17 03:31:41.848988 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.848951    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-token\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.856890 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.856868    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx2lh\" (UniqueName: \"kubernetes.io/projected/740bc52a-9b17-4fe7-ba69-fe88ab56706d-kube-api-access-jx2lh\") pod \"e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") " pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.870234 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.870186    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"]
Sep 17 03:31:41.927035 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.926857    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-workload-certs\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.927035 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.926912    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648fc366-4335-4342-9d41-81e446da0dd2-istio-token\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:41.927035 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.926942    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm9z9\" (UniqueName: \"kubernetes.io/projected/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-kube-api-access-dm9z9\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.927035 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.926978    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-workload-socket\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:41.927418 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.927039    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-workload-certs\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:41.927418 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.927097    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648fc366-4335-4342-9d41-81e446da0dd2-istiod-ca-cert\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:41.927418 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.927125    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-istio-envoy\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:41.927418 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.927174    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istiod-ca-cert\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.927418 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.927231    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-envoy\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.927418 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.927266    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-credential-socket\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:41.927418 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.927383    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-credential-socket\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.927779 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.927524    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:41.927779 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.927634    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-workload-certs\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.927779 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.927761    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-credential-socket\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.927972 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.927837    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-token\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.927972 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.927888    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzp8k\" (UniqueName: \"kubernetes.io/projected/648fc366-4335-4342-9d41-81e446da0dd2-kube-api-access-tzp8k\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:41.928105 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.928083    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648fc366-4335-4342-9d41-81e446da0dd2-istio-podinfo\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:41.928176 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.928155    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-data\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.928291 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.928221    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-workload-socket\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.928291 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.928249    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istiod-ca-cert\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.928291 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.928258    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-podinfo\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.928511 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.928304    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-istio-data\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:41.928511 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.928464    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-data\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.928616 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.928596    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-workload-socket\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.929938 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.929905    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-envoy\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.930905 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.930882    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-podinfo\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.936381 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.936358    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm9z9\" (UniqueName: \"kubernetes.io/projected/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-kube-api-access-dm9z9\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.936894 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.936872    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-token\") pod \"e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") " pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:41.999634 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:41.999599    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:42.034213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.030459    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzp8k\" (UniqueName: \"kubernetes.io/projected/648fc366-4335-4342-9d41-81e446da0dd2-kube-api-access-tzp8k\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.034213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.030536    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648fc366-4335-4342-9d41-81e446da0dd2-istio-podinfo\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.034213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.030600    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-istio-data\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.034213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.030637    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648fc366-4335-4342-9d41-81e446da0dd2-istio-token\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.034213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.030673    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-workload-socket\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.034213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.030710    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-workload-certs\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.034213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.030742    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648fc366-4335-4342-9d41-81e446da0dd2-istiod-ca-cert\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.034213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.030768    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-istio-envoy\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.034213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.030833    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-credential-socket\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.034213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.031265    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-credential-socket\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.034213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.032247    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-workload-socket\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.034213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.032481    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-istio-data\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.034213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.033326    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648fc366-4335-4342-9d41-81e446da0dd2-istiod-ca-cert\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.034213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.033555    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-workload-certs\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.049187 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.047330    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648fc366-4335-4342-9d41-81e446da0dd2-istio-podinfo\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.049187 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.048065    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-istio-envoy\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.052138 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.052089    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648fc366-4335-4342-9d41-81e446da0dd2-istio-token\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.053094 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.053073    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzp8k\" (UniqueName: \"kubernetes.io/projected/648fc366-4335-4342-9d41-81e446da0dd2-kube-api-access-tzp8k\") pod \"e2e-inf-a-297324-openshift-default-5fd877d844-g49lz\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") " pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.170684 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.170658    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:42.208242 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.208187    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"]
Sep 17 03:31:42.209806 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:31:42.209783    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740bc52a_9b17_4fe7_ba69_fe88ab56706d.slice/crio-e415e73ca731d1378bf2b2b7e0bb4866cb70d036ea3c6dce762a98f79eabb22c WatchSource:0}: Error finding container e415e73ca731d1378bf2b2b7e0bb4866cb70d036ea3c6dce762a98f79eabb22c: Status 404 returned error can't find the container with id e415e73ca731d1378bf2b2b7e0bb4866cb70d036ea3c6dce762a98f79eabb22c
Sep 17 03:31:42.212691 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.212650    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:31:42.213111 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.213086    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:31:42.213299 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.213273    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:31:42.308339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.308313    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"]
Sep 17 03:31:42.311156 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:31:42.311133    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc11b8f_129f_48f6_b2c8_fbc4b4ca1b94.slice/crio-c334f564e4ecf12660a14d9db9be33e996cb117329c1f80622eed65238bead60 WatchSource:0}: Error finding container c334f564e4ecf12660a14d9db9be33e996cb117329c1f80622eed65238bead60: Status 404 returned error can't find the container with id c334f564e4ecf12660a14d9db9be33e996cb117329c1f80622eed65238bead60
Sep 17 03:31:42.313636 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.313601    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:31:42.313739 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.313680    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:31:42.313739 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.313726    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:31:42.456355 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.456178    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"]
Sep 17 03:31:42.459403 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:31:42.459373    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648fc366_4335_4342_9d41_81e446da0dd2.slice/crio-f0c2c0d7be763d296ece8b7501c3c7eb1bfa506244a467206ac4d7924a2808a8 WatchSource:0}: Error finding container f0c2c0d7be763d296ece8b7501c3c7eb1bfa506244a467206ac4d7924a2808a8: Status 404 returned error can't find the container with id f0c2c0d7be763d296ece8b7501c3c7eb1bfa506244a467206ac4d7924a2808a8
Sep 17 03:31:42.461952 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.461472    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:31:42.461952 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.461534    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:31:42.461952 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.461562    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:31:42.555905 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.555443    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf" event={"ID":"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94","Type":"ContainerStarted","Data":"d3cc3ab52306b191727104f4d1392040c6fd411795a4a86250da2f3af299f728"}
Sep 17 03:31:42.555905 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.555508    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf" event={"ID":"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94","Type":"ContainerStarted","Data":"c334f564e4ecf12660a14d9db9be33e996cb117329c1f80622eed65238bead60"}
Sep 17 03:31:42.558181 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.558156    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz" event={"ID":"648fc366-4335-4342-9d41-81e446da0dd2","Type":"ContainerStarted","Data":"175953a3d6fbd8b092fa5bfa0ced93fbffcab2ab1c362255695492ddddb3aaab"}
Sep 17 03:31:42.558397 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.558377    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz" event={"ID":"648fc366-4335-4342-9d41-81e446da0dd2","Type":"ContainerStarted","Data":"f0c2c0d7be763d296ece8b7501c3c7eb1bfa506244a467206ac4d7924a2808a8"}
Sep 17 03:31:42.560334 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.560309    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74" event={"ID":"740bc52a-9b17-4fe7-ba69-fe88ab56706d","Type":"ContainerStarted","Data":"db2d98eba7405344e57693995ae8b20ea5366c2e278932cecce4baf9b194eec2"}
Sep 17 03:31:42.560893 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.560872    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74" event={"ID":"740bc52a-9b17-4fe7-ba69-fe88ab56706d","Type":"ContainerStarted","Data":"e415e73ca731d1378bf2b2b7e0bb4866cb70d036ea3c6dce762a98f79eabb22c"}
Sep 17 03:31:42.604773 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.604669    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf" podStartSLOduration=1.604620378 podStartE2EDuration="1.604620378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:31:42.583151739 +0000 UTC m=+753.186247746" watchObservedRunningTime="2026-09-17 03:31:42.604620378 +0000 UTC m=+753.207716385"
Sep 17 03:31:42.606208 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.606145    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74" podStartSLOduration=1.606133547 podStartE2EDuration="1.606133547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:31:42.602304557 +0000 UTC m=+753.205400564" watchObservedRunningTime="2026-09-17 03:31:42.606133547 +0000 UTC m=+753.209229552"
Sep 17 03:31:42.623597 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.623547    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz" podStartSLOduration=1.623529196 podStartE2EDuration="1.623529196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:31:42.620419682 +0000 UTC m=+753.223515689" watchObservedRunningTime="2026-09-17 03:31:42.623529196 +0000 UTC m=+753.226625204"
Sep 17 03:31:42.927990 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:42.927961    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:43.001787 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:43.001749    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:43.006568 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:43.006543    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:43.171681 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:43.171644    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:43.176840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:43.176818    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:43.567146 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:43.567114    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:43.567408 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:43.567381    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:43.567884 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:43.567863    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:31:43.568110 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:43.568092    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:31:43.933748 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:43.933721    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:44.573994 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:44.573965    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:44.574665 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:44.574646    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:31:47.043423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.043392    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"]
Sep 17 03:31:47.047465 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.047441    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"
Sep 17 03:31:47.051445 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.051409    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-297324-dockercfg-g6zhd\""
Sep 17 03:31:47.051683 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.051664    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-297324\""
Sep 17 03:31:47.061410 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.061385    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"]
Sep 17 03:31:47.187966 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.187909    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcx4p\" (UniqueName: \"kubernetes.io/projected/9cc5af42-e9d1-4741-89c1-5b742a035b4d-kube-api-access-pcx4p\") pod \"payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk\" (UID: \"9cc5af42-e9d1-4741-89c1-5b742a035b4d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"
Sep 17 03:31:47.188158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.187997    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc5af42-e9d1-4741-89c1-5b742a035b4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk\" (UID: \"9cc5af42-e9d1-4741-89c1-5b742a035b4d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"
Sep 17 03:31:47.289643 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.289611    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcx4p\" (UniqueName: \"kubernetes.io/projected/9cc5af42-e9d1-4741-89c1-5b742a035b4d-kube-api-access-pcx4p\") pod \"payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk\" (UID: \"9cc5af42-e9d1-4741-89c1-5b742a035b4d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"
Sep 17 03:31:47.289853 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.289654    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc5af42-e9d1-4741-89c1-5b742a035b4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk\" (UID: \"9cc5af42-e9d1-4741-89c1-5b742a035b4d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"
Sep 17 03:31:47.290383 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.290363    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc5af42-e9d1-4741-89c1-5b742a035b4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk\" (UID: \"9cc5af42-e9d1-4741-89c1-5b742a035b4d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"
Sep 17 03:31:47.298720 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.298668    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcx4p\" (UniqueName: \"kubernetes.io/projected/9cc5af42-e9d1-4741-89c1-5b742a035b4d-kube-api-access-pcx4p\") pod \"payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk\" (UID: \"9cc5af42-e9d1-4741-89c1-5b742a035b4d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"
Sep 17 03:31:47.360760 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.360727    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"
Sep 17 03:31:47.500099 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.500064    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"]
Sep 17 03:31:47.513599 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.513569    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"]
Sep 17 03:31:47.513741 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.513693    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:47.515991 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.515870    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-214fea-serving-cert\""
Sep 17 03:31:47.642392 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.642367    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"]
Sep 17 03:31:47.644111 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:31:47.644087    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc5af42_e9d1_4741_89c1_5b742a035b4d.slice/crio-7e4469d2d4e09f1340505bb29d33a1fb49b2ec71985b45f0bfb1e0a9aa188e1f WatchSource:0}: Error finding container 7e4469d2d4e09f1340505bb29d33a1fb49b2ec71985b45f0bfb1e0a9aa188e1f: Status 404 returned error can't find the container with id 7e4469d2d4e09f1340505bb29d33a1fb49b2ec71985b45f0bfb1e0a9aa188e1f
Sep 17 03:31:47.692587 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.692564    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d524126a-97d5-4888-ad42-d850ab79e8a5-maas-api-tls\") pod \"maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:47.692720 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.692634    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6swk\" (UniqueName: \"kubernetes.io/projected/d524126a-97d5-4888-ad42-d850ab79e8a5-kube-api-access-q6swk\") pod \"maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:47.692720 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.692673    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d524126a-97d5-4888-ad42-d850ab79e8a5-metrics-certs\") pod \"maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:47.692720 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.692707    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d524126a-97d5-4888-ad42-d850ab79e8a5-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:47.793466 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.793435    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6swk\" (UniqueName: \"kubernetes.io/projected/d524126a-97d5-4888-ad42-d850ab79e8a5-kube-api-access-q6swk\") pod \"maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:47.793630 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.793507    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d524126a-97d5-4888-ad42-d850ab79e8a5-metrics-certs\") pod \"maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:47.793675 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.793651    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d524126a-97d5-4888-ad42-d850ab79e8a5-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:47.793717 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.793674    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d524126a-97d5-4888-ad42-d850ab79e8a5-maas-api-tls\") pod \"maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:47.794290 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.794266    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d524126a-97d5-4888-ad42-d850ab79e8a5-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:47.795898 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.795866    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d524126a-97d5-4888-ad42-d850ab79e8a5-metrics-certs\") pod \"maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:47.796006 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.795931    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d524126a-97d5-4888-ad42-d850ab79e8a5-maas-api-tls\") pod \"maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:47.801683 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.801662    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6swk\" (UniqueName: \"kubernetes.io/projected/d524126a-97d5-4888-ad42-d850ab79e8a5-kube-api-access-q6swk\") pod \"maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:47.827243 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:47.827173    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:48.106954 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:48.106874    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"]
Sep 17 03:31:48.111571 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:31:48.111538    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd524126a_97d5_4888_ad42_d850ab79e8a5.slice/crio-4b8bd17f1ad248bdbb6ec53200b76aa4e09b9d097c8fea5ed1281fd1f0009099 WatchSource:0}: Error finding container 4b8bd17f1ad248bdbb6ec53200b76aa4e09b9d097c8fea5ed1281fd1f0009099: Status 404 returned error can't find the container with id 4b8bd17f1ad248bdbb6ec53200b76aa4e09b9d097c8fea5ed1281fd1f0009099
Sep 17 03:31:48.599924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:48.599531    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw" event={"ID":"d524126a-97d5-4888-ad42-d850ab79e8a5","Type":"ContainerStarted","Data":"9a148efdb0d1955b1ff1755e34fdd8cb182f2b2af77db58dbdb23425657a76eb"}
Sep 17 03:31:48.599924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:48.599603    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw" event={"ID":"d524126a-97d5-4888-ad42-d850ab79e8a5","Type":"ContainerStarted","Data":"4b8bd17f1ad248bdbb6ec53200b76aa4e09b9d097c8fea5ed1281fd1f0009099"}
Sep 17 03:31:48.599924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:48.599765    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:48.602250 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:48.602219    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk" event={"ID":"9cc5af42-e9d1-4741-89c1-5b742a035b4d","Type":"ContainerStarted","Data":"69ca6406c45efa50a2ab102492d52b5cf5bf4c50c9bfb8444c8a8e6eb3dc4b50"}
Sep 17 03:31:48.602338 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:48.602265    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk" event={"ID":"9cc5af42-e9d1-4741-89c1-5b742a035b4d","Type":"ContainerStarted","Data":"7e4469d2d4e09f1340505bb29d33a1fb49b2ec71985b45f0bfb1e0a9aa188e1f"}
Sep 17 03:31:48.602338 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:48.602300    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"
Sep 17 03:31:48.626937 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:48.626895    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw" podStartSLOduration=1.238545803 podStartE2EDuration="1.626876682s" totalImagesPullingTime="388.330879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:31:48.623741513 +0000 UTC m=+759.226837515" watchObservedRunningTime="2026-09-17 03:31:48.626876682 +0000 UTC m=+759.229972686"
Sep 17 03:31:48.646992 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:48.646946    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk" podStartSLOduration=1.646934582 podStartE2EDuration="1.646934582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:31:48.645784772 +0000 UTC m=+759.248880777" watchObservedRunningTime="2026-09-17 03:31:48.646934582 +0000 UTC m=+759.250030585"
Sep 17 03:31:51.152759 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.152712    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"]
Sep 17 03:31:51.156703 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.156682    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.158930 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.158886    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-f6a774-openshift-default-dockercfg-nzgx9\""
Sep 17 03:31:51.169573 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.167680    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"]
Sep 17 03:31:51.228411 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.228372    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7025aadb-3ef1-42af-b67c-b4004b19dffe-istiod-ca-cert\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.228566 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.228415    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-envoy\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.228566 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.228440    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-token\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.228566 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.228474    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-data\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.228566 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.228535    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2tpx\" (UniqueName: \"kubernetes.io/projected/7025aadb-3ef1-42af-b67c-b4004b19dffe-kube-api-access-f2tpx\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.228766 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.228612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-podinfo\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.228766 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.228662    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-workload-certs\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.228766 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.228723    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-workload-socket\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.228766 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.228746    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-credential-socket\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.330224 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.330153    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-workload-certs\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.330419 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.330246    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-workload-socket\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.330419 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.330283    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-credential-socket\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.330419 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.330354    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7025aadb-3ef1-42af-b67c-b4004b19dffe-istiod-ca-cert\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.330419 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.330380    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-envoy\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.330638 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.330428    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-token\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.330638 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.330484    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-data\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.330638 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.330615    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-workload-certs\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.330638 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.330626    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-workload-socket\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.330860 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.330521    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2tpx\" (UniqueName: \"kubernetes.io/projected/7025aadb-3ef1-42af-b67c-b4004b19dffe-kube-api-access-f2tpx\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.330924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.330860    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-data\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.330982 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.330936    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-podinfo\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.331039 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.330986    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-credential-socket\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.331464 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.331441    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7025aadb-3ef1-42af-b67c-b4004b19dffe-istiod-ca-cert\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.333239 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.333214    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-envoy\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.333794 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.333774    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-podinfo\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.338438 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.338412    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2tpx\" (UniqueName: \"kubernetes.io/projected/7025aadb-3ef1-42af-b67c-b4004b19dffe-kube-api-access-f2tpx\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.338922 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.338903    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-token\") pod \"e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") " pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.472372 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.472293    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:51.758118 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.758094    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"]
Sep 17 03:31:51.760448 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:31:51.760418    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7025aadb_3ef1_42af_b67c_b4004b19dffe.slice/crio-2a7fd660b32a2c5fde892e0df4223db373b2f030e133832a8508f85ec9ff820e WatchSource:0}: Error finding container 2a7fd660b32a2c5fde892e0df4223db373b2f030e133832a8508f85ec9ff820e: Status 404 returned error can't find the container with id 2a7fd660b32a2c5fde892e0df4223db373b2f030e133832a8508f85ec9ff820e
Sep 17 03:31:51.763389 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.763352    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:31:51.763477 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.763432    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:31:51.763542 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:51.763526    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:31:52.625335 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:52.625302    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4" event={"ID":"7025aadb-3ef1-42af-b67c-b4004b19dffe","Type":"ContainerStarted","Data":"4b4e44e173b9b688c57c9eb7710e1d057b4f863c5f0bf0852e60bf7c47cf8b3f"}
Sep 17 03:31:52.625783 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:52.625352    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4" event={"ID":"7025aadb-3ef1-42af-b67c-b4004b19dffe","Type":"ContainerStarted","Data":"2a7fd660b32a2c5fde892e0df4223db373b2f030e133832a8508f85ec9ff820e"}
Sep 17 03:31:52.646700 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:52.646653    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4" podStartSLOduration=1.646638826 podStartE2EDuration="1.646638826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:31:52.644078753 +0000 UTC m=+763.247174978" watchObservedRunningTime="2026-09-17 03:31:52.646638826 +0000 UTC m=+763.249734830"
Sep 17 03:31:53.472773 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:53.472739    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:53.478012 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:53.477987    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:53.632616 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:53.632588    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:53.633470 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:53.633446    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:31:54.617391 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:54.617356    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:31:59.610516 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:31:59.610487    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"
Sep 17 03:32:01.043219 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:01.042338    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"]
Sep 17 03:32:01.052211 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:01.051732    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"
Sep 17 03:32:01.055580 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:01.055552    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-f6a774\""
Sep 17 03:32:01.059495 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:01.059473    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"]
Sep 17 03:32:01.164358 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:01.164071    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkbxm\" (UniqueName: \"kubernetes.io/projected/2b806d1d-5bcc-4607-94c1-4a8ecff47c50-kube-api-access-vkbxm\") pod \"payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms\" (UID: \"2b806d1d-5bcc-4607-94c1-4a8ecff47c50\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"
Sep 17 03:32:01.164358 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:01.164148    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b806d1d-5bcc-4607-94c1-4a8ecff47c50-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms\" (UID: \"2b806d1d-5bcc-4607-94c1-4a8ecff47c50\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"
Sep 17 03:32:01.266121 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:01.266018    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbxm\" (UniqueName: \"kubernetes.io/projected/2b806d1d-5bcc-4607-94c1-4a8ecff47c50-kube-api-access-vkbxm\") pod \"payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms\" (UID: \"2b806d1d-5bcc-4607-94c1-4a8ecff47c50\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"
Sep 17 03:32:01.266121 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:01.266101    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b806d1d-5bcc-4607-94c1-4a8ecff47c50-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms\" (UID: \"2b806d1d-5bcc-4607-94c1-4a8ecff47c50\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"
Sep 17 03:32:01.267185 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:01.267155    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b806d1d-5bcc-4607-94c1-4a8ecff47c50-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms\" (UID: \"2b806d1d-5bcc-4607-94c1-4a8ecff47c50\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"
Sep 17 03:32:01.274046 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:01.274021    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbxm\" (UniqueName: \"kubernetes.io/projected/2b806d1d-5bcc-4607-94c1-4a8ecff47c50-kube-api-access-vkbxm\") pod \"payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms\" (UID: \"2b806d1d-5bcc-4607-94c1-4a8ecff47c50\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"
Sep 17 03:32:01.367148 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:01.367054    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"
Sep 17 03:32:01.697162 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:01.697067    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"]
Sep 17 03:32:01.713428 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:01.713364    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms" event={"ID":"2b806d1d-5bcc-4607-94c1-4a8ecff47c50","Type":"ContainerStarted","Data":"da738ce91f038f18823b0437ae714a8f5fa4b2fe78209a234a181fce549669b0"}
Sep 17 03:32:02.263154 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.263120    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"]
Sep 17 03:32:02.267209 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.267174    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"
Sep 17 03:32:02.269766 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.269576    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c0b7e751-dockercfg-d7m9z\""
Sep 17 03:32:02.269766 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.269620    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c0b7e751\""
Sep 17 03:32:02.279399 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.279372    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"]
Sep 17 03:32:02.303968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.303938    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"]
Sep 17 03:32:02.308272 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.308254    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"
Sep 17 03:32:02.323734 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.323643    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"]
Sep 17 03:32:02.381944 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.381882    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d3860a-1abc-4361-ba1a-c9588e446444-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v\" (UID: \"a1d3860a-1abc-4361-ba1a-c9588e446444\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"
Sep 17 03:32:02.382209 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.382019    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62q7h\" (UniqueName: \"kubernetes.io/projected/a1d3860a-1abc-4361-ba1a-c9588e446444-kube-api-access-62q7h\") pod \"payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v\" (UID: \"a1d3860a-1abc-4361-ba1a-c9588e446444\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"
Sep 17 03:32:02.382209 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.382060    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909e9878-25b1-4a73-a9d4-994b59fbc407-plugins-config-volume\") pod \"payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n\" (UID: \"909e9878-25b1-4a73-a9d4-994b59fbc407\") " pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"
Sep 17 03:32:02.382209 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.382086    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jct4\" (UniqueName: \"kubernetes.io/projected/909e9878-25b1-4a73-a9d4-994b59fbc407-kube-api-access-9jct4\") pod \"payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n\" (UID: \"909e9878-25b1-4a73-a9d4-994b59fbc407\") " pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"
Sep 17 03:32:02.483500 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.483461    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d3860a-1abc-4361-ba1a-c9588e446444-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v\" (UID: \"a1d3860a-1abc-4361-ba1a-c9588e446444\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"
Sep 17 03:32:02.483692 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.483569    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62q7h\" (UniqueName: \"kubernetes.io/projected/a1d3860a-1abc-4361-ba1a-c9588e446444-kube-api-access-62q7h\") pod \"payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v\" (UID: \"a1d3860a-1abc-4361-ba1a-c9588e446444\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"
Sep 17 03:32:02.483692 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.483604    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909e9878-25b1-4a73-a9d4-994b59fbc407-plugins-config-volume\") pod \"payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n\" (UID: \"909e9878-25b1-4a73-a9d4-994b59fbc407\") " pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"
Sep 17 03:32:02.483692 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.483628    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jct4\" (UniqueName: \"kubernetes.io/projected/909e9878-25b1-4a73-a9d4-994b59fbc407-kube-api-access-9jct4\") pod \"payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n\" (UID: \"909e9878-25b1-4a73-a9d4-994b59fbc407\") " pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"
Sep 17 03:32:02.484829 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.484798    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909e9878-25b1-4a73-a9d4-994b59fbc407-plugins-config-volume\") pod \"payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n\" (UID: \"909e9878-25b1-4a73-a9d4-994b59fbc407\") " pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"
Sep 17 03:32:02.484951 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.484927    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d3860a-1abc-4361-ba1a-c9588e446444-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v\" (UID: \"a1d3860a-1abc-4361-ba1a-c9588e446444\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"
Sep 17 03:32:02.497922 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.497874    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jct4\" (UniqueName: \"kubernetes.io/projected/909e9878-25b1-4a73-a9d4-994b59fbc407-kube-api-access-9jct4\") pod \"payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n\" (UID: \"909e9878-25b1-4a73-a9d4-994b59fbc407\") " pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"
Sep 17 03:32:02.498481 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.498459    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62q7h\" (UniqueName: \"kubernetes.io/projected/a1d3860a-1abc-4361-ba1a-c9588e446444-kube-api-access-62q7h\") pod \"payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v\" (UID: \"a1d3860a-1abc-4361-ba1a-c9588e446444\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"
Sep 17 03:32:02.581468 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.581379    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"
Sep 17 03:32:02.622387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.622350    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"
Sep 17 03:32:02.745668 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.745093    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms" event={"ID":"2b806d1d-5bcc-4607-94c1-4a8ecff47c50","Type":"ContainerStarted","Data":"e5699918e0b08a568b82acc8e2787dde5c3dab24bebc4cecf2b06655d644409d"}
Sep 17 03:32:02.745668 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.745363    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"
Sep 17 03:32:02.770224 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.767295    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms" podStartSLOduration=1.767273868 podStartE2EDuration="1.767273868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:32:02.764034913 +0000 UTC m=+773.367130919" watchObservedRunningTime="2026-09-17 03:32:02.767273868 +0000 UTC m=+773.370369872"
Sep 17 03:32:02.876014 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.875260    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"]
Sep 17 03:32:02.878489 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:32:02.878449    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909e9878_25b1_4a73_a9d4_994b59fbc407.slice/crio-b611a0f4fd3ec7fff95502943da4e77a3c968596aedf98ad01527b374ccaff57 WatchSource:0}: Error finding container b611a0f4fd3ec7fff95502943da4e77a3c968596aedf98ad01527b374ccaff57: Status 404 returned error can't find the container with id b611a0f4fd3ec7fff95502943da4e77a3c968596aedf98ad01527b374ccaff57
Sep 17 03:32:02.939268 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:02.938169    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"]
Sep 17 03:32:02.940977 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:32:02.940949    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d3860a_1abc_4361_ba1a_c9588e446444.slice/crio-078cef1c7f71f2122fd4d0c5ec76692d4df3d586e6775ad2b56aed068acc2d28 WatchSource:0}: Error finding container 078cef1c7f71f2122fd4d0c5ec76692d4df3d586e6775ad2b56aed068acc2d28: Status 404 returned error can't find the container with id 078cef1c7f71f2122fd4d0c5ec76692d4df3d586e6775ad2b56aed068acc2d28
Sep 17 03:32:03.763794 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:03.763752    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v" event={"ID":"a1d3860a-1abc-4361-ba1a-c9588e446444","Type":"ContainerStarted","Data":"25a5761ca1f39492db457e25b8da6e16eab737ad8ba8afd8b6814e0908475d8f"}
Sep 17 03:32:03.764259 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:03.763837    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"
Sep 17 03:32:03.764259 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:03.763860    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v" event={"ID":"a1d3860a-1abc-4361-ba1a-c9588e446444","Type":"ContainerStarted","Data":"078cef1c7f71f2122fd4d0c5ec76692d4df3d586e6775ad2b56aed068acc2d28"}
Sep 17 03:32:03.769209 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:03.767599    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n" event={"ID":"909e9878-25b1-4a73-a9d4-994b59fbc407","Type":"ContainerStarted","Data":"3e4053cbb30cdc2f8378b9e15be8f7b0eb2e6735cb72617f5f76f5b1891737bf"}
Sep 17 03:32:03.769209 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:03.767668    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n" event={"ID":"909e9878-25b1-4a73-a9d4-994b59fbc407","Type":"ContainerStarted","Data":"b611a0f4fd3ec7fff95502943da4e77a3c968596aedf98ad01527b374ccaff57"}
Sep 17 03:32:03.781573 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:03.781528    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v" podStartSLOduration=1.781511176 podStartE2EDuration="1.781511176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:32:03.780958596 +0000 UTC m=+774.384054604" watchObservedRunningTime="2026-09-17 03:32:03.781511176 +0000 UTC m=+774.384607184"
Sep 17 03:32:03.801391 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:03.801339    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n" podStartSLOduration=1.8013198190000002 podStartE2EDuration="1.801319819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:32:03.797788275 +0000 UTC m=+774.400884273" watchObservedRunningTime="2026-09-17 03:32:03.801319819 +0000 UTC m=+774.404415825"
Sep 17 03:32:04.774774 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:04.774731    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"
Sep 17 03:32:05.789304 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:05.783899    2557 generic.go:356] "Generic (PLEG): container finished" podID="648fc366-4335-4342-9d41-81e446da0dd2" containerID="175953a3d6fbd8b092fa5bfa0ced93fbffcab2ab1c362255695492ddddb3aaab" exitCode=0
Sep 17 03:32:05.789304 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:05.783969    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz" event={"ID":"648fc366-4335-4342-9d41-81e446da0dd2","Type":"ContainerDied","Data":"175953a3d6fbd8b092fa5bfa0ced93fbffcab2ab1c362255695492ddddb3aaab"}
Sep 17 03:32:05.789304 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:05.784527    2557 scope.go:117] "RemoveContainer" containerID="175953a3d6fbd8b092fa5bfa0ced93fbffcab2ab1c362255695492ddddb3aaab"
Sep 17 03:32:05.789304 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:05.785731    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:32:05.789304 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:05.785813    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:32:05.789304 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:05.785853    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:32:06.790692 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:06.790665    2557 generic.go:356] "Generic (PLEG): container finished" podID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerID="d3cc3ab52306b191727104f4d1392040c6fd411795a4a86250da2f3af299f728" exitCode=0
Sep 17 03:32:06.791172 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:06.790744    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf" event={"ID":"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94","Type":"ContainerDied","Data":"d3cc3ab52306b191727104f4d1392040c6fd411795a4a86250da2f3af299f728"}
Sep 17 03:32:06.791311 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:06.791297    2557 scope.go:117] "RemoveContainer" containerID="d3cc3ab52306b191727104f4d1392040c6fd411795a4a86250da2f3af299f728"
Sep 17 03:32:06.792589 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:06.792554    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:32:06.792678 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:06.792640    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:32:06.792722 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:06.792704    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:32:06.792916 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:06.792896    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz" event={"ID":"648fc366-4335-4342-9d41-81e446da0dd2","Type":"ContainerStarted","Data":"7bc7bd3a33071bfc2ddbe96a1c6d2f8c2d38c15fb82c09488e096f38200d8a34"}
Sep 17 03:32:06.794626 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:06.794604    2557 generic.go:356] "Generic (PLEG): container finished" podID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerID="db2d98eba7405344e57693995ae8b20ea5366c2e278932cecce4baf9b194eec2" exitCode=0
Sep 17 03:32:06.794739 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:06.794649    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74" event={"ID":"740bc52a-9b17-4fe7-ba69-fe88ab56706d","Type":"ContainerDied","Data":"db2d98eba7405344e57693995ae8b20ea5366c2e278932cecce4baf9b194eec2"}
Sep 17 03:32:06.795084 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:06.795067    2557 scope.go:117] "RemoveContainer" containerID="db2d98eba7405344e57693995ae8b20ea5366c2e278932cecce4baf9b194eec2"
Sep 17 03:32:06.798270 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:06.796633    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:32:06.798270 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:06.796708    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:32:06.798270 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:06.796751    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:32:07.171379 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:07.171342    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:32:07.171379 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:07.171379    2557 patch_prober.go:28] interesting pod/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 17 03:32:07.171644 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:07.171422    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz" podUID="648fc366-4335-4342-9d41-81e446da0dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 17 03:32:07.802581 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:07.802549    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf" event={"ID":"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94","Type":"ContainerStarted","Data":"bd85ddf58d29bde173534d84e9336befc59d9c1fb505e6fd8e1c7ba9da0aca10"}
Sep 17 03:32:07.804668 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:07.804639    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74" event={"ID":"740bc52a-9b17-4fe7-ba69-fe88ab56706d","Type":"ContainerStarted","Data":"ee8fa69d8eb077b01bbb902a26de07cf5f2ef01652a2cc75a21aef02a649b411"}
Sep 17 03:32:07.928598 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:07.928559    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:32:07.928809 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:07.928686    2557 patch_prober.go:28] interesting pod/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 17 03:32:07.928809 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:07.928732    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74" podUID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 17 03:32:08.000916 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:08.000876    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:32:08.001334 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:08.001305    2557 patch_prober.go:28] interesting pod/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 17 03:32:08.001459 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:08.001359    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf" podUID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 17 03:32:08.171165 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:08.171117    2557 patch_prober.go:28] interesting pod/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 17 03:32:08.171393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:08.171229    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz" podUID="648fc366-4335-4342-9d41-81e446da0dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 17 03:32:08.933610 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:08.933563    2557 patch_prober.go:28] interesting pod/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 17 03:32:08.934132 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:08.933636    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74" podUID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 17 03:32:09.001876 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:09.001833    2557 patch_prober.go:28] interesting pod/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 17 03:32:09.002049 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:09.001894    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf" podUID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 17 03:32:09.172089 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:09.171635    2557 patch_prober.go:28] interesting pod/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 17 03:32:09.172089 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:09.171698    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz" podUID="648fc366-4335-4342-9d41-81e446da0dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 17 03:32:09.943277 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:09.943100    2557 patch_prober.go:28] interesting pod/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 17 03:32:09.943717 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:09.943267    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74" podUID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 17 03:32:10.001748 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:10.001700    2557 patch_prober.go:28] interesting pod/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 17 03:32:10.002046 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:10.001993    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf" podUID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 17 03:32:10.171453 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:10.171410    2557 patch_prober.go:28] interesting pod/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 17 03:32:10.171711 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:10.171480    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz" podUID="648fc366-4335-4342-9d41-81e446da0dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 17 03:32:10.942217 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:10.941609    2557 patch_prober.go:28] interesting pod/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 17 03:32:10.942217 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:10.941677    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74" podUID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 17 03:32:11.003583 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:11.003545    2557 patch_prober.go:28] interesting pod/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 17 03:32:11.004512 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:11.004109    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf" podUID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 17 03:32:11.171742 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:11.171705    2557 patch_prober.go:28] interesting pod/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 17 03:32:11.172017 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:11.171984    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz" podUID="648fc366-4335-4342-9d41-81e446da0dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 17 03:32:11.933219 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:11.932589    2557 patch_prober.go:28] interesting pod/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 17 03:32:11.933219 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:11.932651    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74" podUID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 17 03:32:12.001789 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:12.001741    2557 patch_prober.go:28] interesting pod/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 17 03:32:12.002004 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:12.001814    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf" podUID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 17 03:32:12.171539 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:12.171494    2557 patch_prober.go:28] interesting pod/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Sep 17 03:32:12.172054 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:12.171564    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz" podUID="648fc366-4335-4342-9d41-81e446da0dd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Sep 17 03:32:12.938599 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:12.938564    2557 patch_prober.go:28] interesting pod/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 17 03:32:12.938904 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:12.938873    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74" podUID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 17 03:32:13.003218 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:13.001218    2557 patch_prober.go:28] interesting pod/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Sep 17 03:32:13.003218 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:13.001280    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf" podUID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Sep 17 03:32:13.187527 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:13.187491    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:32:13.188781 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:13.188698    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:32:13.192412 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:13.192385    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:32:13.567966 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:13.567866    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:32:13.770812 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:13.770777    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"
Sep 17 03:32:13.932218 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:13.931512    2557 patch_prober.go:28] interesting pod/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Sep 17 03:32:13.932218 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:13.931570    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74" podUID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Sep 17 03:32:14.085217 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:14.085139    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:32:14.086898 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:14.086872    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:32:14.573924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:14.573890    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:32:14.777170 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:14.777136    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"
Sep 17 03:32:14.946216 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:14.946076    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:32:14.948773 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:14.948673    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:32:15.786564 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:15.786537    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"
Sep 17 03:32:19.903547 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:19.903510    2557 generic.go:356] "Generic (PLEG): container finished" podID="7025aadb-3ef1-42af-b67c-b4004b19dffe" containerID="4b4e44e173b9b688c57c9eb7710e1d057b4f863c5f0bf0852e60bf7c47cf8b3f" exitCode=0
Sep 17 03:32:19.904065 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:19.903570    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4" event={"ID":"7025aadb-3ef1-42af-b67c-b4004b19dffe","Type":"ContainerDied","Data":"4b4e44e173b9b688c57c9eb7710e1d057b4f863c5f0bf0852e60bf7c47cf8b3f"}
Sep 17 03:32:19.904065 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:19.904031    2557 scope.go:117] "RemoveContainer" containerID="4b4e44e173b9b688c57c9eb7710e1d057b4f863c5f0bf0852e60bf7c47cf8b3f"
Sep 17 03:32:19.905680 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:19.905644    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:32:19.905820 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:19.905726    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:32:19.905820 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:19.905769    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:32:20.911146 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:20.911105    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4" event={"ID":"7025aadb-3ef1-42af-b67c-b4004b19dffe","Type":"ContainerStarted","Data":"506e01e1e304c1da0769b8959d0ab089dfe9fa0bc0c08ad7da1b62e08923f6ae"}
Sep 17 03:32:21.473263 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:21.473219    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:32:21.474578 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:21.474548    2557 patch_prober.go:28] interesting pod/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 17 03:32:21.474713 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:21.474602    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4" podUID="7025aadb-3ef1-42af-b67c-b4004b19dffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 17 03:32:21.582293 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:21.582259    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r"]
Sep 17 03:32:21.586322 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:21.586302    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r"
Sep 17 03:32:21.588541 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:21.588519    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:32:21.593583 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:21.593562    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r"]
Sep 17 03:32:21.707108 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:21.707068    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8fedf7e-2f6a-4eb3-be8f-297d71234848-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r\" (UID: \"b8fedf7e-2f6a-4eb3-be8f-297d71234848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r"
Sep 17 03:32:21.808457 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:21.808359    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8fedf7e-2f6a-4eb3-be8f-297d71234848-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r\" (UID: \"b8fedf7e-2f6a-4eb3-be8f-297d71234848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r"
Sep 17 03:32:21.808991 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:21.808972    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8fedf7e-2f6a-4eb3-be8f-297d71234848-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r\" (UID: \"b8fedf7e-2f6a-4eb3-be8f-297d71234848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r"
Sep 17 03:32:21.899256 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:21.899222    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r"
Sep 17 03:32:22.204728 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:22.204703    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r"]
Sep 17 03:32:22.207077 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:32:22.207045    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8fedf7e_2f6a_4eb3_be8f_297d71234848.slice/crio-442d39c8e19f4fb71221029f1d02eda21168c2c9a2f2d7bd568752d425b0eca9 WatchSource:0}: Error finding container 442d39c8e19f4fb71221029f1d02eda21168c2c9a2f2d7bd568752d425b0eca9: Status 404 returned error can't find the container with id 442d39c8e19f4fb71221029f1d02eda21168c2c9a2f2d7bd568752d425b0eca9
Sep 17 03:32:22.473477 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:22.473388    2557 patch_prober.go:28] interesting pod/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 17 03:32:22.473477 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:22.473434    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4" podUID="7025aadb-3ef1-42af-b67c-b4004b19dffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 17 03:32:22.926573 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:22.926530    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r" event={"ID":"b8fedf7e-2f6a-4eb3-be8f-297d71234848","Type":"ContainerStarted","Data":"442d39c8e19f4fb71221029f1d02eda21168c2c9a2f2d7bd568752d425b0eca9"}
Sep 17 03:32:23.473342 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:23.473305    2557 patch_prober.go:28] interesting pod/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 17 03:32:23.473770 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:23.473367    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4" podUID="7025aadb-3ef1-42af-b67c-b4004b19dffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 17 03:32:23.636179 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:23.635295    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:32:24.473261 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:24.473221    2557 patch_prober.go:28] interesting pod/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 17 03:32:24.473436 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:24.473295    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4" podUID="7025aadb-3ef1-42af-b67c-b4004b19dffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 17 03:32:25.473561 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:25.473481    2557 patch_prober.go:28] interesting pod/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Sep 17 03:32:25.473561 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:25.473534    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4" podUID="7025aadb-3ef1-42af-b67c-b4004b19dffe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Sep 17 03:32:25.969880 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:25.969849    2557 generic.go:356] "Generic (PLEG): container finished" podID="b8fedf7e-2f6a-4eb3-be8f-297d71234848" containerID="9f0eb36e95b521fd49ca18b66d7da9421db92ba841476b2f6f2bd39a93a78b23" exitCode=0
Sep 17 03:32:25.970053 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:25.969934    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r" event={"ID":"b8fedf7e-2f6a-4eb3-be8f-297d71234848","Type":"ContainerDied","Data":"9f0eb36e95b521fd49ca18b66d7da9421db92ba841476b2f6f2bd39a93a78b23"}
Sep 17 03:32:26.476889 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:26.476856    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:32:26.478046 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:26.478024    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:32:27.121040 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:27.121012    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r"
Sep 17 03:32:27.290578 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:27.290503    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8fedf7e-2f6a-4eb3-be8f-297d71234848-maas-api-service-ca\") pod \"b8fedf7e-2f6a-4eb3-be8f-297d71234848\" (UID: \"b8fedf7e-2f6a-4eb3-be8f-297d71234848\") "
Sep 17 03:32:27.290828 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:27.290806    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8fedf7e-2f6a-4eb3-be8f-297d71234848-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b8fedf7e-2f6a-4eb3-be8f-297d71234848" (UID: "b8fedf7e-2f6a-4eb3-be8f-297d71234848"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:32:27.391601 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:27.391566    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8fedf7e-2f6a-4eb3-be8f-297d71234848-maas-api-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:32:27.989731 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:27.989696    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r" event={"ID":"b8fedf7e-2f6a-4eb3-be8f-297d71234848","Type":"ContainerDied","Data":"442d39c8e19f4fb71221029f1d02eda21168c2c9a2f2d7bd568752d425b0eca9"}
Sep 17 03:32:27.989731 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:27.989714    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r"
Sep 17 03:32:27.990182 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:27.989775    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="442d39c8e19f4fb71221029f1d02eda21168c2c9a2f2d7bd568752d425b0eca9"
Sep 17 03:32:38.513269 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.513036    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"]
Sep 17 03:32:38.513694 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.513317    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n" podUID="909e9878-25b1-4a73-a9d4-994b59fbc407" containerName="payload-processing" containerID="cri-o://3e4053cbb30cdc2f8378b9e15be8f7b0eb2e6735cb72617f5f76f5b1891737bf" gracePeriod=30
Sep 17 03:32:38.518393 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.518366    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"]
Sep 17 03:32:38.518652 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.518626    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v" podUID="a1d3860a-1abc-4361-ba1a-c9588e446444" containerName="payload-pre-processing" containerID="cri-o://25a5761ca1f39492db457e25b8da6e16eab737ad8ba8afd8b6814e0908475d8f" gracePeriod=30
Sep 17 03:32:38.658582 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.658548    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r"]
Sep 17 03:32:38.660965 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.660934    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c0b7e751-qbl7r"]
Sep 17 03:32:38.865681 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.865645    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"
Sep 17 03:32:38.868778 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.868757    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"
Sep 17 03:32:38.990551 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.990488    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62q7h\" (UniqueName: \"kubernetes.io/projected/a1d3860a-1abc-4361-ba1a-c9588e446444-kube-api-access-62q7h\") pod \"a1d3860a-1abc-4361-ba1a-c9588e446444\" (UID: \"a1d3860a-1abc-4361-ba1a-c9588e446444\") "
Sep 17 03:32:38.990799 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.990617    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909e9878-25b1-4a73-a9d4-994b59fbc407-plugins-config-volume\") pod \"909e9878-25b1-4a73-a9d4-994b59fbc407\" (UID: \"909e9878-25b1-4a73-a9d4-994b59fbc407\") "
Sep 17 03:32:38.991468 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.990935    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d3860a-1abc-4361-ba1a-c9588e446444-plugins-config-volume\") pod \"a1d3860a-1abc-4361-ba1a-c9588e446444\" (UID: \"a1d3860a-1abc-4361-ba1a-c9588e446444\") "
Sep 17 03:32:38.991468 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.990998    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jct4\" (UniqueName: \"kubernetes.io/projected/909e9878-25b1-4a73-a9d4-994b59fbc407-kube-api-access-9jct4\") pod \"909e9878-25b1-4a73-a9d4-994b59fbc407\" (UID: \"909e9878-25b1-4a73-a9d4-994b59fbc407\") "
Sep 17 03:32:38.991468 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.991390    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909e9878-25b1-4a73-a9d4-994b59fbc407-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "909e9878-25b1-4a73-a9d4-994b59fbc407" (UID: "909e9878-25b1-4a73-a9d4-994b59fbc407"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:32:38.991739 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.991478    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/909e9878-25b1-4a73-a9d4-994b59fbc407-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:32:38.991739 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.991724    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d3860a-1abc-4361-ba1a-c9588e446444-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1d3860a-1abc-4361-ba1a-c9588e446444" (UID: "a1d3860a-1abc-4361-ba1a-c9588e446444"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:32:38.993683 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.993658    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909e9878-25b1-4a73-a9d4-994b59fbc407-kube-api-access-9jct4" (OuterVolumeSpecName: "kube-api-access-9jct4") pod "909e9878-25b1-4a73-a9d4-994b59fbc407" (UID: "909e9878-25b1-4a73-a9d4-994b59fbc407"). InnerVolumeSpecName "kube-api-access-9jct4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:32:38.993760 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:38.993679    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d3860a-1abc-4361-ba1a-c9588e446444-kube-api-access-62q7h" (OuterVolumeSpecName: "kube-api-access-62q7h") pod "a1d3860a-1abc-4361-ba1a-c9588e446444" (UID: "a1d3860a-1abc-4361-ba1a-c9588e446444"). InnerVolumeSpecName "kube-api-access-62q7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:32:39.065675 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.065644    2557 generic.go:356] "Generic (PLEG): container finished" podID="a1d3860a-1abc-4361-ba1a-c9588e446444" containerID="25a5761ca1f39492db457e25b8da6e16eab737ad8ba8afd8b6814e0908475d8f" exitCode=0
Sep 17 03:32:39.065949 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.065928    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v" event={"ID":"a1d3860a-1abc-4361-ba1a-c9588e446444","Type":"ContainerDied","Data":"25a5761ca1f39492db457e25b8da6e16eab737ad8ba8afd8b6814e0908475d8f"}
Sep 17 03:32:39.066148 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.066123    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v" event={"ID":"a1d3860a-1abc-4361-ba1a-c9588e446444","Type":"ContainerDied","Data":"078cef1c7f71f2122fd4d0c5ec76692d4df3d586e6775ad2b56aed068acc2d28"}
Sep 17 03:32:39.066303 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.066289    2557 scope.go:117] "RemoveContainer" containerID="25a5761ca1f39492db457e25b8da6e16eab737ad8ba8afd8b6814e0908475d8f"
Sep 17 03:32:39.066602 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.066586    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"
Sep 17 03:32:39.070264 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.070238    2557 generic.go:356] "Generic (PLEG): container finished" podID="909e9878-25b1-4a73-a9d4-994b59fbc407" containerID="3e4053cbb30cdc2f8378b9e15be8f7b0eb2e6735cb72617f5f76f5b1891737bf" exitCode=0
Sep 17 03:32:39.070361 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.070328    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n" event={"ID":"909e9878-25b1-4a73-a9d4-994b59fbc407","Type":"ContainerDied","Data":"3e4053cbb30cdc2f8378b9e15be8f7b0eb2e6735cb72617f5f76f5b1891737bf"}
Sep 17 03:32:39.070434 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.070368    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n" event={"ID":"909e9878-25b1-4a73-a9d4-994b59fbc407","Type":"ContainerDied","Data":"b611a0f4fd3ec7fff95502943da4e77a3c968596aedf98ad01527b374ccaff57"}
Sep 17 03:32:39.070496 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.070436    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"
Sep 17 03:32:39.091749 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.090500    2557 scope.go:117] "RemoveContainer" containerID="25a5761ca1f39492db457e25b8da6e16eab737ad8ba8afd8b6814e0908475d8f"
Sep 17 03:32:39.091749 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:32:39.090849    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a5761ca1f39492db457e25b8da6e16eab737ad8ba8afd8b6814e0908475d8f\": container with ID starting with 25a5761ca1f39492db457e25b8da6e16eab737ad8ba8afd8b6814e0908475d8f not found: ID does not exist" containerID="25a5761ca1f39492db457e25b8da6e16eab737ad8ba8afd8b6814e0908475d8f"
Sep 17 03:32:39.091749 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.090879    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a5761ca1f39492db457e25b8da6e16eab737ad8ba8afd8b6814e0908475d8f"} err="failed to get container status \"25a5761ca1f39492db457e25b8da6e16eab737ad8ba8afd8b6814e0908475d8f\": rpc error: code = NotFound desc = could not find container \"25a5761ca1f39492db457e25b8da6e16eab737ad8ba8afd8b6814e0908475d8f\": container with ID starting with 25a5761ca1f39492db457e25b8da6e16eab737ad8ba8afd8b6814e0908475d8f not found: ID does not exist"
Sep 17 03:32:39.091749 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.090900    2557 scope.go:117] "RemoveContainer" containerID="3e4053cbb30cdc2f8378b9e15be8f7b0eb2e6735cb72617f5f76f5b1891737bf"
Sep 17 03:32:39.092557 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.092533    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62q7h\" (UniqueName: \"kubernetes.io/projected/a1d3860a-1abc-4361-ba1a-c9588e446444-kube-api-access-62q7h\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:32:39.092673 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.092562    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d3860a-1abc-4361-ba1a-c9588e446444-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:32:39.092673 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.092577    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jct4\" (UniqueName: \"kubernetes.io/projected/909e9878-25b1-4a73-a9d4-994b59fbc407-kube-api-access-9jct4\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:32:39.111613 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.111574    2557 scope.go:117] "RemoveContainer" containerID="3e4053cbb30cdc2f8378b9e15be8f7b0eb2e6735cb72617f5f76f5b1891737bf"
Sep 17 03:32:39.112423 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:32:39.112103    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4053cbb30cdc2f8378b9e15be8f7b0eb2e6735cb72617f5f76f5b1891737bf\": container with ID starting with 3e4053cbb30cdc2f8378b9e15be8f7b0eb2e6735cb72617f5f76f5b1891737bf not found: ID does not exist" containerID="3e4053cbb30cdc2f8378b9e15be8f7b0eb2e6735cb72617f5f76f5b1891737bf"
Sep 17 03:32:39.112423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.112143    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4053cbb30cdc2f8378b9e15be8f7b0eb2e6735cb72617f5f76f5b1891737bf"} err="failed to get container status \"3e4053cbb30cdc2f8378b9e15be8f7b0eb2e6735cb72617f5f76f5b1891737bf\": rpc error: code = NotFound desc = could not find container \"3e4053cbb30cdc2f8378b9e15be8f7b0eb2e6735cb72617f5f76f5b1891737bf\": container with ID starting with 3e4053cbb30cdc2f8378b9e15be8f7b0eb2e6735cb72617f5f76f5b1891737bf not found: ID does not exist"
Sep 17 03:32:39.115625 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.115602    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"]
Sep 17 03:32:39.118045 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.118021    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c0b7e751-746c4f6787-qmt5n"]
Sep 17 03:32:39.127609 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.127592    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"]
Sep 17 03:32:39.135591 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.135569    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c0b7e751-7dfcfc9fd6-qbg7v"]
Sep 17 03:32:39.917610 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.917574    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="909e9878-25b1-4a73-a9d4-994b59fbc407" path="/var/lib/kubelet/pods/909e9878-25b1-4a73-a9d4-994b59fbc407/volumes"
Sep 17 03:32:39.918013 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.917985    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1d3860a-1abc-4361-ba1a-c9588e446444" path="/var/lib/kubelet/pods/a1d3860a-1abc-4361-ba1a-c9588e446444/volumes"
Sep 17 03:32:39.918399 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:39.918379    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8fedf7e-2f6a-4eb3-be8f-297d71234848" path="/var/lib/kubelet/pods/b8fedf7e-2f6a-4eb3-be8f-297d71234848/volumes"
Sep 17 03:32:44.206021 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.205987    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"]
Sep 17 03:32:44.206676 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.206455    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8fedf7e-2f6a-4eb3-be8f-297d71234848" containerName="revoke-keys"
Sep 17 03:32:44.206676 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.206467    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8fedf7e-2f6a-4eb3-be8f-297d71234848" containerName="revoke-keys"
Sep 17 03:32:44.206676 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.206478    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="909e9878-25b1-4a73-a9d4-994b59fbc407" containerName="payload-processing"
Sep 17 03:32:44.206676 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.206484    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="909e9878-25b1-4a73-a9d4-994b59fbc407" containerName="payload-processing"
Sep 17 03:32:44.206676 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.206502    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d3860a-1abc-4361-ba1a-c9588e446444" containerName="payload-pre-processing"
Sep 17 03:32:44.206676 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.206508    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d3860a-1abc-4361-ba1a-c9588e446444" containerName="payload-pre-processing"
Sep 17 03:32:44.206676 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.206565    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8fedf7e-2f6a-4eb3-be8f-297d71234848" containerName="revoke-keys"
Sep 17 03:32:44.206676 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.206573    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="909e9878-25b1-4a73-a9d4-994b59fbc407" containerName="payload-processing"
Sep 17 03:32:44.206676 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.206579    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1d3860a-1abc-4361-ba1a-c9588e446444" containerName="payload-pre-processing"
Sep 17 03:32:44.211064 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.211042    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:44.213661 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.213635    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-0ae566a4-serving-cert\""
Sep 17 03:32:44.220865 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.220841    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"]
Sep 17 03:32:44.242264 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.242237    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eefbbe98-8176-4661-9ae6-2a71935fd3c3-metrics-certs\") pod \"maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:44.242403 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.242306    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eefbbe98-8176-4661-9ae6-2a71935fd3c3-maas-api-tls\") pod \"maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:44.242474 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.242401    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eefbbe98-8176-4661-9ae6-2a71935fd3c3-openshift-service-ca\") pod \"maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:44.242548 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.242525    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9pxk\" (UniqueName: \"kubernetes.io/projected/eefbbe98-8176-4661-9ae6-2a71935fd3c3-kube-api-access-g9pxk\") pod \"maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:44.343709 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.343676    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eefbbe98-8176-4661-9ae6-2a71935fd3c3-maas-api-tls\") pod \"maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:44.343897 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.343740    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eefbbe98-8176-4661-9ae6-2a71935fd3c3-openshift-service-ca\") pod \"maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:44.343897 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.343850    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9pxk\" (UniqueName: \"kubernetes.io/projected/eefbbe98-8176-4661-9ae6-2a71935fd3c3-kube-api-access-g9pxk\") pod \"maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:44.344022 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.343938    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eefbbe98-8176-4661-9ae6-2a71935fd3c3-metrics-certs\") pod \"maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:44.344752 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.344722    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eefbbe98-8176-4661-9ae6-2a71935fd3c3-openshift-service-ca\") pod \"maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:44.346094 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.346073    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eefbbe98-8176-4661-9ae6-2a71935fd3c3-maas-api-tls\") pod \"maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:44.346377 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.346354    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eefbbe98-8176-4661-9ae6-2a71935fd3c3-metrics-certs\") pod \"maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:44.351503 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.351479    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9pxk\" (UniqueName: \"kubernetes.io/projected/eefbbe98-8176-4661-9ae6-2a71935fd3c3-kube-api-access-g9pxk\") pod \"maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:44.525313 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.525219    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:44.817971 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:44.817946    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"]
Sep 17 03:32:44.819338 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:32:44.819305    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeefbbe98_8176_4661_9ae6_2a71935fd3c3.slice/crio-f793dd785c74da05200e254c7fab445a0401b06b81342b187301a1bf57782c84 WatchSource:0}: Error finding container f793dd785c74da05200e254c7fab445a0401b06b81342b187301a1bf57782c84: Status 404 returned error can't find the container with id f793dd785c74da05200e254c7fab445a0401b06b81342b187301a1bf57782c84
Sep 17 03:32:45.048176 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:45.048147    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"]
Sep 17 03:32:45.052947 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:45.052927    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"
Sep 17 03:32:45.056412 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:45.056382    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-0ae566a4\""
Sep 17 03:32:45.056528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:45.056501    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-0ae566a4-dockercfg-qx5r4\""
Sep 17 03:32:45.063003 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:45.062979    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"]
Sep 17 03:32:45.119701 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:45.119662    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch" event={"ID":"eefbbe98-8176-4661-9ae6-2a71935fd3c3","Type":"ContainerStarted","Data":"f793dd785c74da05200e254c7fab445a0401b06b81342b187301a1bf57782c84"}
Sep 17 03:32:45.152841 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:45.152806    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb9jv\" (UniqueName: \"kubernetes.io/projected/5ad2bf67-a4be-440c-848a-feb1ee422235-kube-api-access-cb9jv\") pod \"payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v\" (UID: \"5ad2bf67-a4be-440c-848a-feb1ee422235\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"
Sep 17 03:32:45.153011 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:45.152989    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad2bf67-a4be-440c-848a-feb1ee422235-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v\" (UID: \"5ad2bf67-a4be-440c-848a-feb1ee422235\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"
Sep 17 03:32:45.253649 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:45.253614    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad2bf67-a4be-440c-848a-feb1ee422235-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v\" (UID: \"5ad2bf67-a4be-440c-848a-feb1ee422235\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"
Sep 17 03:32:45.254013 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:45.253664    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb9jv\" (UniqueName: \"kubernetes.io/projected/5ad2bf67-a4be-440c-848a-feb1ee422235-kube-api-access-cb9jv\") pod \"payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v\" (UID: \"5ad2bf67-a4be-440c-848a-feb1ee422235\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"
Sep 17 03:32:45.254386 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:45.254361    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad2bf67-a4be-440c-848a-feb1ee422235-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v\" (UID: \"5ad2bf67-a4be-440c-848a-feb1ee422235\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"
Sep 17 03:32:45.261985 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:45.261966    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb9jv\" (UniqueName: \"kubernetes.io/projected/5ad2bf67-a4be-440c-848a-feb1ee422235-kube-api-access-cb9jv\") pod \"payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v\" (UID: \"5ad2bf67-a4be-440c-848a-feb1ee422235\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"
Sep 17 03:32:45.366812 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:45.366790    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"
Sep 17 03:32:45.655298 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:45.655229    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"]
Sep 17 03:32:45.658061 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:32:45.658037    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad2bf67_a4be_440c_848a_feb1ee422235.slice/crio-97c73ad260b38d5eafe97a3a435e417bf0eee5f593849ce29ac36b9430cd8767 WatchSource:0}: Error finding container 97c73ad260b38d5eafe97a3a435e417bf0eee5f593849ce29ac36b9430cd8767: Status 404 returned error can't find the container with id 97c73ad260b38d5eafe97a3a435e417bf0eee5f593849ce29ac36b9430cd8767
Sep 17 03:32:46.127785 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:46.127685    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch" event={"ID":"eefbbe98-8176-4661-9ae6-2a71935fd3c3","Type":"ContainerStarted","Data":"3421a719b92948287b86b7edcc18b81b923e4ea59fee86aa68ee992c902dff6d"}
Sep 17 03:32:46.127785 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:46.127753    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:46.129681 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:46.129651    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v" event={"ID":"5ad2bf67-a4be-440c-848a-feb1ee422235","Type":"ContainerStarted","Data":"2b3b29fb0dc869b44271ce7a4f4a5c0c617c038940df0bfe2c07beec2acb7764"}
Sep 17 03:32:46.129837 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:46.129705    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v" event={"ID":"5ad2bf67-a4be-440c-848a-feb1ee422235","Type":"ContainerStarted","Data":"97c73ad260b38d5eafe97a3a435e417bf0eee5f593849ce29ac36b9430cd8767"}
Sep 17 03:32:46.129837 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:46.129737    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"
Sep 17 03:32:46.148377 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:46.148327    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch" podStartSLOduration=1.709242606 podStartE2EDuration="2.148308776s" totalImagesPullingTime="439.06617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:32:46.14730365 +0000 UTC m=+816.750399654" watchObservedRunningTime="2026-09-17 03:32:46.148308776 +0000 UTC m=+816.751404772"
Sep 17 03:32:46.166377 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:46.166337    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v" podStartSLOduration=1.166324917 podStartE2EDuration="1.166324917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:32:46.164714961 +0000 UTC m=+816.767810977" watchObservedRunningTime="2026-09-17 03:32:46.166324917 +0000 UTC m=+816.769420921"
Sep 17 03:32:52.140898 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:52.140871    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:32:57.137713 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:32:57.137684    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"
Sep 17 03:33:14.299634 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.299592    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"]
Sep 17 03:33:14.300144 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.299847    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch" podUID="eefbbe98-8176-4661-9ae6-2a71935fd3c3" containerName="maas-api" containerID="cri-o://3421a719b92948287b86b7edcc18b81b923e4ea59fee86aa68ee992c902dff6d" gracePeriod=30
Sep 17 03:33:14.370740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.370709    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"]
Sep 17 03:33:14.370965 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.370934    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v" podUID="5ad2bf67-a4be-440c-848a-feb1ee422235" containerName="payload-pre-processing" containerID="cri-o://2b3b29fb0dc869b44271ce7a4f4a5c0c617c038940df0bfe2c07beec2acb7764" gracePeriod=30
Sep 17 03:33:14.684791 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.684518    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:33:14.694091 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.694017    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"
Sep 17 03:33:14.838848 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.837671    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eefbbe98-8176-4661-9ae6-2a71935fd3c3-openshift-service-ca\") pod \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") "
Sep 17 03:33:14.838848 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.837716    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eefbbe98-8176-4661-9ae6-2a71935fd3c3-metrics-certs\") pod \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") "
Sep 17 03:33:14.838848 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.837766    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad2bf67-a4be-440c-848a-feb1ee422235-plugins-config-volume\") pod \"5ad2bf67-a4be-440c-848a-feb1ee422235\" (UID: \"5ad2bf67-a4be-440c-848a-feb1ee422235\") "
Sep 17 03:33:14.838848 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.837896    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9pxk\" (UniqueName: \"kubernetes.io/projected/eefbbe98-8176-4661-9ae6-2a71935fd3c3-kube-api-access-g9pxk\") pod \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") "
Sep 17 03:33:14.838848 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.837933    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb9jv\" (UniqueName: \"kubernetes.io/projected/5ad2bf67-a4be-440c-848a-feb1ee422235-kube-api-access-cb9jv\") pod \"5ad2bf67-a4be-440c-848a-feb1ee422235\" (UID: \"5ad2bf67-a4be-440c-848a-feb1ee422235\") "
Sep 17 03:33:14.838848 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.838007    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eefbbe98-8176-4661-9ae6-2a71935fd3c3-maas-api-tls\") pod \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\" (UID: \"eefbbe98-8176-4661-9ae6-2a71935fd3c3\") "
Sep 17 03:33:14.839798 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.839765    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eefbbe98-8176-4661-9ae6-2a71935fd3c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eefbbe98-8176-4661-9ae6-2a71935fd3c3" (UID: "eefbbe98-8176-4661-9ae6-2a71935fd3c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:33:14.839899 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.839841    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad2bf67-a4be-440c-848a-feb1ee422235-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ad2bf67-a4be-440c-848a-feb1ee422235" (UID: "5ad2bf67-a4be-440c-848a-feb1ee422235"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:33:14.842728 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.842691    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefbbe98-8176-4661-9ae6-2a71935fd3c3-kube-api-access-g9pxk" (OuterVolumeSpecName: "kube-api-access-g9pxk") pod "eefbbe98-8176-4661-9ae6-2a71935fd3c3" (UID: "eefbbe98-8176-4661-9ae6-2a71935fd3c3"). InnerVolumeSpecName "kube-api-access-g9pxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:33:14.843048 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.843026    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad2bf67-a4be-440c-848a-feb1ee422235-kube-api-access-cb9jv" (OuterVolumeSpecName: "kube-api-access-cb9jv") pod "5ad2bf67-a4be-440c-848a-feb1ee422235" (UID: "5ad2bf67-a4be-440c-848a-feb1ee422235"). InnerVolumeSpecName "kube-api-access-cb9jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:33:14.843330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.843304    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eefbbe98-8176-4661-9ae6-2a71935fd3c3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eefbbe98-8176-4661-9ae6-2a71935fd3c3" (UID: "eefbbe98-8176-4661-9ae6-2a71935fd3c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:33:14.843709 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.843677    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eefbbe98-8176-4661-9ae6-2a71935fd3c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eefbbe98-8176-4661-9ae6-2a71935fd3c3" (UID: "eefbbe98-8176-4661-9ae6-2a71935fd3c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:33:14.939766 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.939733    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eefbbe98-8176-4661-9ae6-2a71935fd3c3-openshift-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:14.939766 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.939760    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eefbbe98-8176-4661-9ae6-2a71935fd3c3-metrics-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:14.939766 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.939770    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ad2bf67-a4be-440c-848a-feb1ee422235-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:14.939962 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.939779    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9pxk\" (UniqueName: \"kubernetes.io/projected/eefbbe98-8176-4661-9ae6-2a71935fd3c3-kube-api-access-g9pxk\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:14.939962 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.939788    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb9jv\" (UniqueName: \"kubernetes.io/projected/5ad2bf67-a4be-440c-848a-feb1ee422235-kube-api-access-cb9jv\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:14.939962 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:14.939796    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eefbbe98-8176-4661-9ae6-2a71935fd3c3-maas-api-tls\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:15.289312 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.289257    2557 generic.go:356] "Generic (PLEG): container finished" podID="5ad2bf67-a4be-440c-848a-feb1ee422235" containerID="2b3b29fb0dc869b44271ce7a4f4a5c0c617c038940df0bfe2c07beec2acb7764" exitCode=0
Sep 17 03:33:15.289312 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.289308    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"
Sep 17 03:33:15.289770 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.289344    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v" event={"ID":"5ad2bf67-a4be-440c-848a-feb1ee422235","Type":"ContainerDied","Data":"2b3b29fb0dc869b44271ce7a4f4a5c0c617c038940df0bfe2c07beec2acb7764"}
Sep 17 03:33:15.289770 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.289412    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v" event={"ID":"5ad2bf67-a4be-440c-848a-feb1ee422235","Type":"ContainerDied","Data":"97c73ad260b38d5eafe97a3a435e417bf0eee5f593849ce29ac36b9430cd8767"}
Sep 17 03:33:15.289770 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.289449    2557 scope.go:117] "RemoveContainer" containerID="2b3b29fb0dc869b44271ce7a4f4a5c0c617c038940df0bfe2c07beec2acb7764"
Sep 17 03:33:15.291454 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.291431    2557 generic.go:356] "Generic (PLEG): container finished" podID="eefbbe98-8176-4661-9ae6-2a71935fd3c3" containerID="3421a719b92948287b86b7edcc18b81b923e4ea59fee86aa68ee992c902dff6d" exitCode=0
Sep 17 03:33:15.291560 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.291484    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"
Sep 17 03:33:15.291560 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.291514    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch" event={"ID":"eefbbe98-8176-4661-9ae6-2a71935fd3c3","Type":"ContainerDied","Data":"3421a719b92948287b86b7edcc18b81b923e4ea59fee86aa68ee992c902dff6d"}
Sep 17 03:33:15.291651 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.291562    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch" event={"ID":"eefbbe98-8176-4661-9ae6-2a71935fd3c3","Type":"ContainerDied","Data":"f793dd785c74da05200e254c7fab445a0401b06b81342b187301a1bf57782c84"}
Sep 17 03:33:15.299542 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.299519    2557 scope.go:117] "RemoveContainer" containerID="2b3b29fb0dc869b44271ce7a4f4a5c0c617c038940df0bfe2c07beec2acb7764"
Sep 17 03:33:15.299807 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:33:15.299787    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3b29fb0dc869b44271ce7a4f4a5c0c617c038940df0bfe2c07beec2acb7764\": container with ID starting with 2b3b29fb0dc869b44271ce7a4f4a5c0c617c038940df0bfe2c07beec2acb7764 not found: ID does not exist" containerID="2b3b29fb0dc869b44271ce7a4f4a5c0c617c038940df0bfe2c07beec2acb7764"
Sep 17 03:33:15.300133 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.299813    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3b29fb0dc869b44271ce7a4f4a5c0c617c038940df0bfe2c07beec2acb7764"} err="failed to get container status \"2b3b29fb0dc869b44271ce7a4f4a5c0c617c038940df0bfe2c07beec2acb7764\": rpc error: code = NotFound desc = could not find container \"2b3b29fb0dc869b44271ce7a4f4a5c0c617c038940df0bfe2c07beec2acb7764\": container with ID starting with 2b3b29fb0dc869b44271ce7a4f4a5c0c617c038940df0bfe2c07beec2acb7764 not found: ID does not exist"
Sep 17 03:33:15.300133 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.299829    2557 scope.go:117] "RemoveContainer" containerID="3421a719b92948287b86b7edcc18b81b923e4ea59fee86aa68ee992c902dff6d"
Sep 17 03:33:15.308997 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.308694    2557 scope.go:117] "RemoveContainer" containerID="3421a719b92948287b86b7edcc18b81b923e4ea59fee86aa68ee992c902dff6d"
Sep 17 03:33:15.309079 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:33:15.308993    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3421a719b92948287b86b7edcc18b81b923e4ea59fee86aa68ee992c902dff6d\": container with ID starting with 3421a719b92948287b86b7edcc18b81b923e4ea59fee86aa68ee992c902dff6d not found: ID does not exist" containerID="3421a719b92948287b86b7edcc18b81b923e4ea59fee86aa68ee992c902dff6d"
Sep 17 03:33:15.309079 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.309014    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3421a719b92948287b86b7edcc18b81b923e4ea59fee86aa68ee992c902dff6d"} err="failed to get container status \"3421a719b92948287b86b7edcc18b81b923e4ea59fee86aa68ee992c902dff6d\": rpc error: code = NotFound desc = could not find container \"3421a719b92948287b86b7edcc18b81b923e4ea59fee86aa68ee992c902dff6d\": container with ID starting with 3421a719b92948287b86b7edcc18b81b923e4ea59fee86aa68ee992c902dff6d not found: ID does not exist"
Sep 17 03:33:15.315072 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.315049    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"]
Sep 17 03:33:15.323291 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.323268    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0ae566a4-5879b6c67d-8w99v"]
Sep 17 03:33:15.335230 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.335180    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"]
Sep 17 03:33:15.340999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.340980    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0ae566a4-55d4b6f474-6qqch"]
Sep 17 03:33:15.413975 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.413951    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"]
Sep 17 03:33:15.414456 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.414442    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eefbbe98-8176-4661-9ae6-2a71935fd3c3" containerName="maas-api"
Sep 17 03:33:15.414517 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.414457    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="eefbbe98-8176-4661-9ae6-2a71935fd3c3" containerName="maas-api"
Sep 17 03:33:15.414517 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.414471    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ad2bf67-a4be-440c-848a-feb1ee422235" containerName="payload-pre-processing"
Sep 17 03:33:15.414517 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.414477    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ad2bf67-a4be-440c-848a-feb1ee422235" containerName="payload-pre-processing"
Sep 17 03:33:15.414615 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.414549    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="eefbbe98-8176-4661-9ae6-2a71935fd3c3" containerName="maas-api"
Sep 17 03:33:15.414615 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.414558    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ad2bf67-a4be-440c-848a-feb1ee422235" containerName="payload-pre-processing"
Sep 17 03:33:15.418960 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.418934    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.423744 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.423725    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3b91ce2e-openshift-default-dockercfg-q6t2z\""
Sep 17 03:33:15.450156 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.450131    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"]
Sep 17 03:33:15.545637 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.545537    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-data\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.545637 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.545588    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-podinfo\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.545842 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.545699    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-workload-certs\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.545842 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.545778    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-envoy\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.545842 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.545809    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-workload-socket\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.545956 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.545912    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkr2c\" (UniqueName: \"kubernetes.io/projected/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-kube-api-access-wkr2c\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.545994 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.545978    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istiod-ca-cert\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.546031 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.546015    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-credential-socket\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.546066 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.546048    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-token\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.646826 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.646797    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-token\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.647000 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.646846    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-data\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.647000 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.646871    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-podinfo\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.647000 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.646893    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-workload-certs\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.647000 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.646926    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-envoy\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.647233 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.647081    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-workload-socket\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.647302 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.647226    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkr2c\" (UniqueName: \"kubernetes.io/projected/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-kube-api-access-wkr2c\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.647302 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.647278    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istiod-ca-cert\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.647418 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.647316    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-credential-socket\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.647473 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.647430    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-workload-certs\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.647525 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.647463    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-data\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.647576 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.647534    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-workload-socket\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.647694 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.647675    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-credential-socket\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.647995 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.647979    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istiod-ca-cert\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.649257 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.649235    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-envoy\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.649487 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.649469    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-podinfo\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.654547 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.654527    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-token\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.654833 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.654812    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkr2c\" (UniqueName: \"kubernetes.io/projected/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-kube-api-access-wkr2c\") pod \"e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") " pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.732600 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.732577    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:15.915817 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.915789    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ad2bf67-a4be-440c-848a-feb1ee422235" path="/var/lib/kubelet/pods/5ad2bf67-a4be-440c-848a-feb1ee422235/volumes"
Sep 17 03:33:15.916234 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:15.916219    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eefbbe98-8176-4661-9ae6-2a71935fd3c3" path="/var/lib/kubelet/pods/eefbbe98-8176-4661-9ae6-2a71935fd3c3/volumes"
Sep 17 03:33:16.021454 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:33:16.017243    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf8660e3_04cf_40e1_9b0f_b5d90ed4c271.slice/crio-303d5d53e1de298dd2120557506163bc2a8a3ec59c74415a99bd89ab742ccecb WatchSource:0}: Error finding container 303d5d53e1de298dd2120557506163bc2a8a3ec59c74415a99bd89ab742ccecb: Status 404 returned error can't find the container with id 303d5d53e1de298dd2120557506163bc2a8a3ec59c74415a99bd89ab742ccecb
Sep 17 03:33:16.021454 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:16.017383    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"]
Sep 17 03:33:16.023802 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:16.023575    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:33:16.023802 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:16.023707    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:33:16.024129 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:16.023869    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:33:16.299397 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:16.299300    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq" event={"ID":"af8660e3-04cf-40e1-9b0f-b5d90ed4c271","Type":"ContainerStarted","Data":"38760a6992359dc46aa52e244542aa62a937fc9f6c3e33dd09369aafb2d422c8"}
Sep 17 03:33:16.299397 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:16.299357    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq" event={"ID":"af8660e3-04cf-40e1-9b0f-b5d90ed4c271","Type":"ContainerStarted","Data":"303d5d53e1de298dd2120557506163bc2a8a3ec59c74415a99bd89ab742ccecb"}
Sep 17 03:33:16.325239 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:16.325174    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq" podStartSLOduration=1.325157925 podStartE2EDuration="1.325157925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:33:16.32320623 +0000 UTC m=+846.926302227" watchObservedRunningTime="2026-09-17 03:33:16.325157925 +0000 UTC m=+846.928253928"
Sep 17 03:33:16.732950 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:16.732918    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:17.739466 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:17.739438    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:18.312528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:18.312500    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:18.313258 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:18.313236    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:21.316436 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:21.316390    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"]
Sep 17 03:33:21.320513 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:21.320491    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"
Sep 17 03:33:21.322604 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:21.322578    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3b91ce2e\""
Sep 17 03:33:21.322737 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:21.322588    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3b91ce2e-dockercfg-fgp9m\""
Sep 17 03:33:21.331497 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:21.331470    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"]
Sep 17 03:33:21.506520 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:21.506485    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdz2l\" (UniqueName: \"kubernetes.io/projected/34ffd997-2f53-4eb7-a844-1cfba5e739de-kube-api-access-bdz2l\") pod \"payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j\" (UID: \"34ffd997-2f53-4eb7-a844-1cfba5e739de\") " pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"
Sep 17 03:33:21.506520 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:21.506519    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ffd997-2f53-4eb7-a844-1cfba5e739de-plugins-config-volume\") pod \"payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j\" (UID: \"34ffd997-2f53-4eb7-a844-1cfba5e739de\") " pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"
Sep 17 03:33:21.607690 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:21.607613    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdz2l\" (UniqueName: \"kubernetes.io/projected/34ffd997-2f53-4eb7-a844-1cfba5e739de-kube-api-access-bdz2l\") pod \"payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j\" (UID: \"34ffd997-2f53-4eb7-a844-1cfba5e739de\") " pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"
Sep 17 03:33:21.607690 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:21.607651    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ffd997-2f53-4eb7-a844-1cfba5e739de-plugins-config-volume\") pod \"payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j\" (UID: \"34ffd997-2f53-4eb7-a844-1cfba5e739de\") " pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"
Sep 17 03:33:21.608377 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:21.608359    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ffd997-2f53-4eb7-a844-1cfba5e739de-plugins-config-volume\") pod \"payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j\" (UID: \"34ffd997-2f53-4eb7-a844-1cfba5e739de\") " pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"
Sep 17 03:33:21.619443 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:21.619423    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdz2l\" (UniqueName: \"kubernetes.io/projected/34ffd997-2f53-4eb7-a844-1cfba5e739de-kube-api-access-bdz2l\") pod \"payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j\" (UID: \"34ffd997-2f53-4eb7-a844-1cfba5e739de\") " pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"
Sep 17 03:33:21.635352 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:21.635332    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"
Sep 17 03:33:21.914142 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:33:21.914109    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ffd997_2f53_4eb7_a844_1cfba5e739de.slice/crio-e5e780581bc6b213f42be3516eac81e135bd103e994ab0718142b37d572ef362 WatchSource:0}: Error finding container e5e780581bc6b213f42be3516eac81e135bd103e994ab0718142b37d572ef362: Status 404 returned error can't find the container with id e5e780581bc6b213f42be3516eac81e135bd103e994ab0718142b37d572ef362
Sep 17 03:33:21.919620 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:21.919599    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"]
Sep 17 03:33:22.341385 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:22.341294    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j" event={"ID":"34ffd997-2f53-4eb7-a844-1cfba5e739de","Type":"ContainerStarted","Data":"cf5645b71cc145a88785fbc4c7edf9045738f342b17552e1c286e77b44950994"}
Sep 17 03:33:22.341385 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:22.341358    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"
Sep 17 03:33:22.341385 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:22.341376    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j" event={"ID":"34ffd997-2f53-4eb7-a844-1cfba5e739de","Type":"ContainerStarted","Data":"e5e780581bc6b213f42be3516eac81e135bd103e994ab0718142b37d572ef362"}
Sep 17 03:33:22.356836 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:22.356780    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j" podStartSLOduration=1.356761672 podStartE2EDuration="1.356761672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:33:22.355836828 +0000 UTC m=+852.958932836" watchObservedRunningTime="2026-09-17 03:33:22.356761672 +0000 UTC m=+852.959857680"
Sep 17 03:33:33.349786 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:33.349756    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"
Sep 17 03:33:43.427284 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:43.427244    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg"]
Sep 17 03:33:43.431990 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:43.431971    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg"
Sep 17 03:33:43.434706 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:43.434679    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:33:43.437466 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:43.437414    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg"]
Sep 17 03:33:43.519858 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:43.519827    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28a3f7d-af2b-405d-9aae-49f8413839a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg\" (UID: \"d28a3f7d-af2b-405d-9aae-49f8413839a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg"
Sep 17 03:33:43.620839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:43.620806    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28a3f7d-af2b-405d-9aae-49f8413839a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg\" (UID: \"d28a3f7d-af2b-405d-9aae-49f8413839a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg"
Sep 17 03:33:43.621506 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:43.621472    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28a3f7d-af2b-405d-9aae-49f8413839a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg\" (UID: \"d28a3f7d-af2b-405d-9aae-49f8413839a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg"
Sep 17 03:33:43.748409 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:43.748327    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg"
Sep 17 03:33:44.032550 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:44.032526    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg"]
Sep 17 03:33:44.034152 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:33:44.034127    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28a3f7d_af2b_405d_9aae_49f8413839a4.slice/crio-4a4f4a3e233c52eafad8c3a08bd2dce890d1f94949aa3f9a9a99c45425f74625 WatchSource:0}: Error finding container 4a4f4a3e233c52eafad8c3a08bd2dce890d1f94949aa3f9a9a99c45425f74625: Status 404 returned error can't find the container with id 4a4f4a3e233c52eafad8c3a08bd2dce890d1f94949aa3f9a9a99c45425f74625
Sep 17 03:33:44.479457 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:44.479417    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg" event={"ID":"d28a3f7d-af2b-405d-9aae-49f8413839a4","Type":"ContainerStarted","Data":"608ff78affd25e9abc5ab36ee3e3d8d3c5b430dd903b7097ec9896b037c96611"}
Sep 17 03:33:44.479969 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:44.479484    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg" event={"ID":"d28a3f7d-af2b-405d-9aae-49f8413839a4","Type":"ContainerStarted","Data":"4a4f4a3e233c52eafad8c3a08bd2dce890d1f94949aa3f9a9a99c45425f74625"}
Sep 17 03:33:44.481024 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:44.480999    2557 generic.go:356] "Generic (PLEG): container finished" podID="af8660e3-04cf-40e1-9b0f-b5d90ed4c271" containerID="38760a6992359dc46aa52e244542aa62a937fc9f6c3e33dd09369aafb2d422c8" exitCode=0
Sep 17 03:33:44.481110 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:44.481049    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq" event={"ID":"af8660e3-04cf-40e1-9b0f-b5d90ed4c271","Type":"ContainerDied","Data":"38760a6992359dc46aa52e244542aa62a937fc9f6c3e33dd09369aafb2d422c8"}
Sep 17 03:33:44.481432 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:44.481418    2557 scope.go:117] "RemoveContainer" containerID="38760a6992359dc46aa52e244542aa62a937fc9f6c3e33dd09369aafb2d422c8"
Sep 17 03:33:44.482540 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:44.482509    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:33:44.482634 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:44.482577    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:33:44.482634 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:44.482609    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:33:44.496648 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:44.496607    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg" podStartSLOduration=1.49659324 podStartE2EDuration="1.49659324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:33:44.493726815 +0000 UTC m=+875.096822820" watchObservedRunningTime="2026-09-17 03:33:44.49659324 +0000 UTC m=+875.099689244"
Sep 17 03:33:45.488747 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:45.488708    2557 generic.go:356] "Generic (PLEG): container finished" podID="d28a3f7d-af2b-405d-9aae-49f8413839a4" containerID="608ff78affd25e9abc5ab36ee3e3d8d3c5b430dd903b7097ec9896b037c96611" exitCode=0
Sep 17 03:33:45.489122 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:45.488750    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg" event={"ID":"d28a3f7d-af2b-405d-9aae-49f8413839a4","Type":"ContainerDied","Data":"608ff78affd25e9abc5ab36ee3e3d8d3c5b430dd903b7097ec9896b037c96611"}
Sep 17 03:33:45.490818 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:45.490794    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq" event={"ID":"af8660e3-04cf-40e1-9b0f-b5d90ed4c271","Type":"ContainerStarted","Data":"b89621721a20c313c2e00705702eaa5e2c613f94cb0a9b1228ac226d312aa109"}
Sep 17 03:33:45.732928 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:45.732888    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:45.734490 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:45.734461    2557 patch_prober.go:28] interesting pod/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 17 03:33:45.734719 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:45.734694    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq" podUID="af8660e3-04cf-40e1-9b0f-b5d90ed4c271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 17 03:33:46.678672 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:46.678649    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg"
Sep 17 03:33:46.737902 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:46.737504    2557 patch_prober.go:28] interesting pod/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 17 03:33:46.737902 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:46.737559    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq" podUID="af8660e3-04cf-40e1-9b0f-b5d90ed4c271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 17 03:33:46.756108 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:46.756039    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28a3f7d-af2b-405d-9aae-49f8413839a4-maas-api-service-ca\") pod \"d28a3f7d-af2b-405d-9aae-49f8413839a4\" (UID: \"d28a3f7d-af2b-405d-9aae-49f8413839a4\") "
Sep 17 03:33:46.756411 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:46.756379    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28a3f7d-af2b-405d-9aae-49f8413839a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d28a3f7d-af2b-405d-9aae-49f8413839a4" (UID: "d28a3f7d-af2b-405d-9aae-49f8413839a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:33:46.857695 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:46.857660    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28a3f7d-af2b-405d-9aae-49f8413839a4-maas-api-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:47.507027 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:47.506987    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg" event={"ID":"d28a3f7d-af2b-405d-9aae-49f8413839a4","Type":"ContainerDied","Data":"4a4f4a3e233c52eafad8c3a08bd2dce890d1f94949aa3f9a9a99c45425f74625"}
Sep 17 03:33:47.507208 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:47.507064    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a4f4a3e233c52eafad8c3a08bd2dce890d1f94949aa3f9a9a99c45425f74625"
Sep 17 03:33:47.507208 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:47.506999    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg"
Sep 17 03:33:47.733564 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:47.733520    2557 patch_prober.go:28] interesting pod/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 17 03:33:47.734029 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:47.733589    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq" podUID="af8660e3-04cf-40e1-9b0f-b5d90ed4c271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 17 03:33:48.313089 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:48.313060    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:48.733620 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:48.733585    2557 patch_prober.go:28] interesting pod/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 17 03:33:48.734210 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:48.734158    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq" podUID="af8660e3-04cf-40e1-9b0f-b5d90ed4c271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 17 03:33:49.733852 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:49.733809    2557 patch_prober.go:28] interesting pod/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Sep 17 03:33:49.734356 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:49.733876    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq" podUID="af8660e3-04cf-40e1-9b0f-b5d90ed4c271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Sep 17 03:33:50.318775 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:50.318741    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"]
Sep 17 03:33:50.319059 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:50.319003    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j" podUID="34ffd997-2f53-4eb7-a844-1cfba5e739de" containerName="payload-processing" containerID="cri-o://cf5645b71cc145a88785fbc4c7edf9045738f342b17552e1c286e77b44950994" gracePeriod=30
Sep 17 03:33:50.543215 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:50.542533    2557 generic.go:356] "Generic (PLEG): container finished" podID="34ffd997-2f53-4eb7-a844-1cfba5e739de" containerID="cf5645b71cc145a88785fbc4c7edf9045738f342b17552e1c286e77b44950994" exitCode=0
Sep 17 03:33:50.543215 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:50.542595    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j" event={"ID":"34ffd997-2f53-4eb7-a844-1cfba5e739de","Type":"ContainerDied","Data":"cf5645b71cc145a88785fbc4c7edf9045738f342b17552e1c286e77b44950994"}
Sep 17 03:33:50.544820 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:50.544626    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg"]
Sep 17 03:33:50.547315 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:50.547262    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b91ce2e-f72lg"]
Sep 17 03:33:50.634235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:50.634206    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"
Sep 17 03:33:50.696821 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:50.696775    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdz2l\" (UniqueName: \"kubernetes.io/projected/34ffd997-2f53-4eb7-a844-1cfba5e739de-kube-api-access-bdz2l\") pod \"34ffd997-2f53-4eb7-a844-1cfba5e739de\" (UID: \"34ffd997-2f53-4eb7-a844-1cfba5e739de\") "
Sep 17 03:33:50.697052 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:50.696857    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ffd997-2f53-4eb7-a844-1cfba5e739de-plugins-config-volume\") pod \"34ffd997-2f53-4eb7-a844-1cfba5e739de\" (UID: \"34ffd997-2f53-4eb7-a844-1cfba5e739de\") "
Sep 17 03:33:50.698000 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:50.697736    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ffd997-2f53-4eb7-a844-1cfba5e739de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "34ffd997-2f53-4eb7-a844-1cfba5e739de" (UID: "34ffd997-2f53-4eb7-a844-1cfba5e739de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:33:50.700739 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:50.700671    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ffd997-2f53-4eb7-a844-1cfba5e739de-kube-api-access-bdz2l" (OuterVolumeSpecName: "kube-api-access-bdz2l") pod "34ffd997-2f53-4eb7-a844-1cfba5e739de" (UID: "34ffd997-2f53-4eb7-a844-1cfba5e739de"). InnerVolumeSpecName "kube-api-access-bdz2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:33:50.798277 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:50.798249    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdz2l\" (UniqueName: \"kubernetes.io/projected/34ffd997-2f53-4eb7-a844-1cfba5e739de-kube-api-access-bdz2l\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:50.798277 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:50.798276    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ffd997-2f53-4eb7-a844-1cfba5e739de-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:50.915021 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:50.914973    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:50.916652 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:50.916627    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:51.549608 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:51.549577    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"
Sep 17 03:33:51.549806 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:51.549613    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j" event={"ID":"34ffd997-2f53-4eb7-a844-1cfba5e739de","Type":"ContainerDied","Data":"e5e780581bc6b213f42be3516eac81e135bd103e994ab0718142b37d572ef362"}
Sep 17 03:33:51.549806 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:51.549689    2557 scope.go:117] "RemoveContainer" containerID="cf5645b71cc145a88785fbc4c7edf9045738f342b17552e1c286e77b44950994"
Sep 17 03:33:51.572655 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:51.572628    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"]
Sep 17 03:33:51.578551 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:51.578525    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3b91ce2e-57b889d959-dcl5j"]
Sep 17 03:33:51.642879 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:51.642851    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"]
Sep 17 03:33:51.915480 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:51.915445    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34ffd997-2f53-4eb7-a844-1cfba5e739de" path="/var/lib/kubelet/pods/34ffd997-2f53-4eb7-a844-1cfba5e739de/volumes"
Sep 17 03:33:51.915952 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:51.915803    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d28a3f7d-af2b-405d-9aae-49f8413839a4" path="/var/lib/kubelet/pods/d28a3f7d-af2b-405d-9aae-49f8413839a4/volumes"
Sep 17 03:33:52.557026 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:52.556975    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq" podUID="af8660e3-04cf-40e1-9b0f-b5d90ed4c271" containerName="istio-proxy" containerID="cri-o://b89621721a20c313c2e00705702eaa5e2c613f94cb0a9b1228ac226d312aa109" gracePeriod=30
Sep 17 03:33:57.075426 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.075396    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"]
Sep 17 03:33:57.075893 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.075878    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d28a3f7d-af2b-405d-9aae-49f8413839a4" containerName="revoke-keys"
Sep 17 03:33:57.075938 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.075896    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d28a3f7d-af2b-405d-9aae-49f8413839a4" containerName="revoke-keys"
Sep 17 03:33:57.075938 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.075911    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34ffd997-2f53-4eb7-a844-1cfba5e739de" containerName="payload-processing"
Sep 17 03:33:57.075938 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.075917    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="34ffd997-2f53-4eb7-a844-1cfba5e739de" containerName="payload-processing"
Sep 17 03:33:57.076028 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.075985    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="34ffd997-2f53-4eb7-a844-1cfba5e739de" containerName="payload-processing"
Sep 17 03:33:57.076028 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.075996    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d28a3f7d-af2b-405d-9aae-49f8413839a4" containerName="revoke-keys"
Sep 17 03:33:57.080408 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.080378    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"
Sep 17 03:33:57.082663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.082638    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-acf311cd-dockercfg-7htcj\""
Sep 17 03:33:57.082837 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.082818    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-acf311cd\""
Sep 17 03:33:57.090763 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.090745    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"]
Sep 17 03:33:57.122977 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.122951    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"]
Sep 17 03:33:57.127363 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.127343    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"
Sep 17 03:33:57.135865 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.135842    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"]
Sep 17 03:33:57.163591 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.163554    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdzbj\" (UniqueName: \"kubernetes.io/projected/625158c3-9351-4f7d-9e26-cdbde0e0465c-kube-api-access-vdzbj\") pod \"payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct\" (UID: \"625158c3-9351-4f7d-9e26-cdbde0e0465c\") " pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"
Sep 17 03:33:57.163890 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.163868    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625158c3-9351-4f7d-9e26-cdbde0e0465c-plugins-config-volume\") pod \"payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct\" (UID: \"625158c3-9351-4f7d-9e26-cdbde0e0465c\") " pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"
Sep 17 03:33:57.265250 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.265206    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdzbj\" (UniqueName: \"kubernetes.io/projected/625158c3-9351-4f7d-9e26-cdbde0e0465c-kube-api-access-vdzbj\") pod \"payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct\" (UID: \"625158c3-9351-4f7d-9e26-cdbde0e0465c\") " pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"
Sep 17 03:33:57.265250 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.265249    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625158c3-9351-4f7d-9e26-cdbde0e0465c-plugins-config-volume\") pod \"payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct\" (UID: \"625158c3-9351-4f7d-9e26-cdbde0e0465c\") " pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"
Sep 17 03:33:57.265492 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.265294    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1896049-c33c-402a-977e-9944c91a10b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5\" (UID: \"a1896049-c33c-402a-977e-9944c91a10b0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"
Sep 17 03:33:57.265492 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.265332    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnm6z\" (UniqueName: \"kubernetes.io/projected/a1896049-c33c-402a-977e-9944c91a10b0-kube-api-access-wnm6z\") pod \"payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5\" (UID: \"a1896049-c33c-402a-977e-9944c91a10b0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"
Sep 17 03:33:57.266032 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.266006    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625158c3-9351-4f7d-9e26-cdbde0e0465c-plugins-config-volume\") pod \"payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct\" (UID: \"625158c3-9351-4f7d-9e26-cdbde0e0465c\") " pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"
Sep 17 03:33:57.274328 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.274301    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdzbj\" (UniqueName: \"kubernetes.io/projected/625158c3-9351-4f7d-9e26-cdbde0e0465c-kube-api-access-vdzbj\") pod \"payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct\" (UID: \"625158c3-9351-4f7d-9e26-cdbde0e0465c\") " pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"
Sep 17 03:33:57.366776 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.366753    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1896049-c33c-402a-977e-9944c91a10b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5\" (UID: \"a1896049-c33c-402a-977e-9944c91a10b0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"
Sep 17 03:33:57.366868 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.366793    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnm6z\" (UniqueName: \"kubernetes.io/projected/a1896049-c33c-402a-977e-9944c91a10b0-kube-api-access-wnm6z\") pod \"payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5\" (UID: \"a1896049-c33c-402a-977e-9944c91a10b0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"
Sep 17 03:33:57.367392 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.367375    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1896049-c33c-402a-977e-9944c91a10b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5\" (UID: \"a1896049-c33c-402a-977e-9944c91a10b0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"
Sep 17 03:33:57.376857 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.376832    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnm6z\" (UniqueName: \"kubernetes.io/projected/a1896049-c33c-402a-977e-9944c91a10b0-kube-api-access-wnm6z\") pod \"payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5\" (UID: \"a1896049-c33c-402a-977e-9944c91a10b0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"
Sep 17 03:33:57.416403 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.416376    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"
Sep 17 03:33:57.438083 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.438062    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"
Sep 17 03:33:57.689728 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.689706    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"]
Sep 17 03:33:57.691063 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:33:57.691038    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625158c3_9351_4f7d_9e26_cdbde0e0465c.slice/crio-98ce2503ba60a370af450f98b76792acd7637b66d76792a01eb8144cf25d9813 WatchSource:0}: Error finding container 98ce2503ba60a370af450f98b76792acd7637b66d76792a01eb8144cf25d9813: Status 404 returned error can't find the container with id 98ce2503ba60a370af450f98b76792acd7637b66d76792a01eb8144cf25d9813
Sep 17 03:33:57.734627 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.734598    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"]
Sep 17 03:33:57.741387 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:33:57.741335    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1896049_c33c_402a_977e_9944c91a10b0.slice/crio-e30be4101aca9448749e09f6e1d3bb875942b51ddc6ab3d9fd1a51311ff20793 WatchSource:0}: Error finding container e30be4101aca9448749e09f6e1d3bb875942b51ddc6ab3d9fd1a51311ff20793: Status 404 returned error can't find the container with id e30be4101aca9448749e09f6e1d3bb875942b51ddc6ab3d9fd1a51311ff20793
Sep 17 03:33:57.817054 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.817029    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:57.973511 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.973445    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-workload-certs\") pod \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") "
Sep 17 03:33:57.973511 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.973491    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-workload-socket\") pod \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") "
Sep 17 03:33:57.973713 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.973518    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkr2c\" (UniqueName: \"kubernetes.io/projected/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-kube-api-access-wkr2c\") pod \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") "
Sep 17 03:33:57.973713 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.973548    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-data\") pod \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") "
Sep 17 03:33:57.973713 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.973574    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-podinfo\") pod \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") "
Sep 17 03:33:57.973713 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.973600    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-token\") pod \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") "
Sep 17 03:33:57.973921 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.973708    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af8660e3-04cf-40e1-9b0f-b5d90ed4c271" (UID: "af8660e3-04cf-40e1-9b0f-b5d90ed4c271"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:33:57.973921 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.973765    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istiod-ca-cert\") pod \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") "
Sep 17 03:33:57.973921 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.973772    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af8660e3-04cf-40e1-9b0f-b5d90ed4c271" (UID: "af8660e3-04cf-40e1-9b0f-b5d90ed4c271"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:33:57.973921 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.973808    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-envoy\") pod \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") "
Sep 17 03:33:57.973921 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.973855    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-credential-socket\") pod \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\" (UID: \"af8660e3-04cf-40e1-9b0f-b5d90ed4c271\") "
Sep 17 03:33:57.973921 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.973857    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-data" (OuterVolumeSpecName: "istio-data") pod "af8660e3-04cf-40e1-9b0f-b5d90ed4c271" (UID: "af8660e3-04cf-40e1-9b0f-b5d90ed4c271"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:33:57.974269 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.974103    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af8660e3-04cf-40e1-9b0f-b5d90ed4c271" (UID: "af8660e3-04cf-40e1-9b0f-b5d90ed4c271"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:33:57.974269 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.974240    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af8660e3-04cf-40e1-9b0f-b5d90ed4c271" (UID: "af8660e3-04cf-40e1-9b0f-b5d90ed4c271"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:33:57.974467 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.974444    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:57.974537 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.974470    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:57.974537 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.974487    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:57.974537 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.974501    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:57.974537 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.974513    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:57.975902 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.975874    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af8660e3-04cf-40e1-9b0f-b5d90ed4c271" (UID: "af8660e3-04cf-40e1-9b0f-b5d90ed4c271"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:33:57.976005 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.975910    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af8660e3-04cf-40e1-9b0f-b5d90ed4c271" (UID: "af8660e3-04cf-40e1-9b0f-b5d90ed4c271"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:33:57.976158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.976134    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-kube-api-access-wkr2c" (OuterVolumeSpecName: "kube-api-access-wkr2c") pod "af8660e3-04cf-40e1-9b0f-b5d90ed4c271" (UID: "af8660e3-04cf-40e1-9b0f-b5d90ed4c271"). InnerVolumeSpecName "kube-api-access-wkr2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:33:57.976459 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:57.976432    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-token" (OuterVolumeSpecName: "istio-token") pod "af8660e3-04cf-40e1-9b0f-b5d90ed4c271" (UID: "af8660e3-04cf-40e1-9b0f-b5d90ed4c271"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:33:58.075410 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.075383    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:58.075410 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.075404    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkr2c\" (UniqueName: \"kubernetes.io/projected/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-kube-api-access-wkr2c\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:58.075410 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.075413    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:58.075773 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.075422    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af8660e3-04cf-40e1-9b0f-b5d90ed4c271-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:33:58.592455 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.592424    2557 generic.go:356] "Generic (PLEG): container finished" podID="af8660e3-04cf-40e1-9b0f-b5d90ed4c271" containerID="b89621721a20c313c2e00705702eaa5e2c613f94cb0a9b1228ac226d312aa109" exitCode=0
Sep 17 03:33:58.592628 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.592473    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"
Sep 17 03:33:58.592628 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.592509    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq" event={"ID":"af8660e3-04cf-40e1-9b0f-b5d90ed4c271","Type":"ContainerDied","Data":"b89621721a20c313c2e00705702eaa5e2c613f94cb0a9b1228ac226d312aa109"}
Sep 17 03:33:58.592628 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.592564    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq" event={"ID":"af8660e3-04cf-40e1-9b0f-b5d90ed4c271","Type":"ContainerDied","Data":"303d5d53e1de298dd2120557506163bc2a8a3ec59c74415a99bd89ab742ccecb"}
Sep 17 03:33:58.592628 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.592595    2557 scope.go:117] "RemoveContainer" containerID="b89621721a20c313c2e00705702eaa5e2c613f94cb0a9b1228ac226d312aa109"
Sep 17 03:33:58.594261 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.594237    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5" event={"ID":"a1896049-c33c-402a-977e-9944c91a10b0","Type":"ContainerStarted","Data":"dd6aa57d41257285c2a00519c86b08c7652462880c778c3c40bd69ef0109b33a"}
Sep 17 03:33:58.594406 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.594280    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5" event={"ID":"a1896049-c33c-402a-977e-9944c91a10b0","Type":"ContainerStarted","Data":"e30be4101aca9448749e09f6e1d3bb875942b51ddc6ab3d9fd1a51311ff20793"}
Sep 17 03:33:58.594406 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.594306    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"
Sep 17 03:33:58.596099 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.596064    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct" event={"ID":"625158c3-9351-4f7d-9e26-cdbde0e0465c","Type":"ContainerStarted","Data":"a73dd908f6c0a5288aeb53103268aa5f38ebdd0de805f2402cec629230ba070b"}
Sep 17 03:33:58.596295 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.596108    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct" event={"ID":"625158c3-9351-4f7d-9e26-cdbde0e0465c","Type":"ContainerStarted","Data":"98ce2503ba60a370af450f98b76792acd7637b66d76792a01eb8144cf25d9813"}
Sep 17 03:33:58.596295 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.596208    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"
Sep 17 03:33:58.602761 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.602745    2557 scope.go:117] "RemoveContainer" containerID="38760a6992359dc46aa52e244542aa62a937fc9f6c3e33dd09369aafb2d422c8"
Sep 17 03:33:58.611035 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.611022    2557 scope.go:117] "RemoveContainer" containerID="b89621721a20c313c2e00705702eaa5e2c613f94cb0a9b1228ac226d312aa109"
Sep 17 03:33:58.611309 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:33:58.611289    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89621721a20c313c2e00705702eaa5e2c613f94cb0a9b1228ac226d312aa109\": container with ID starting with b89621721a20c313c2e00705702eaa5e2c613f94cb0a9b1228ac226d312aa109 not found: ID does not exist" containerID="b89621721a20c313c2e00705702eaa5e2c613f94cb0a9b1228ac226d312aa109"
Sep 17 03:33:58.611379 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.611316    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89621721a20c313c2e00705702eaa5e2c613f94cb0a9b1228ac226d312aa109"} err="failed to get container status \"b89621721a20c313c2e00705702eaa5e2c613f94cb0a9b1228ac226d312aa109\": rpc error: code = NotFound desc = could not find container \"b89621721a20c313c2e00705702eaa5e2c613f94cb0a9b1228ac226d312aa109\": container with ID starting with b89621721a20c313c2e00705702eaa5e2c613f94cb0a9b1228ac226d312aa109 not found: ID does not exist"
Sep 17 03:33:58.611379 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.611332    2557 scope.go:117] "RemoveContainer" containerID="38760a6992359dc46aa52e244542aa62a937fc9f6c3e33dd09369aafb2d422c8"
Sep 17 03:33:58.611547 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:33:58.611525    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38760a6992359dc46aa52e244542aa62a937fc9f6c3e33dd09369aafb2d422c8\": container with ID starting with 38760a6992359dc46aa52e244542aa62a937fc9f6c3e33dd09369aafb2d422c8 not found: ID does not exist" containerID="38760a6992359dc46aa52e244542aa62a937fc9f6c3e33dd09369aafb2d422c8"
Sep 17 03:33:58.611599 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.611557    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38760a6992359dc46aa52e244542aa62a937fc9f6c3e33dd09369aafb2d422c8"} err="failed to get container status \"38760a6992359dc46aa52e244542aa62a937fc9f6c3e33dd09369aafb2d422c8\": rpc error: code = NotFound desc = could not find container \"38760a6992359dc46aa52e244542aa62a937fc9f6c3e33dd09369aafb2d422c8\": container with ID starting with 38760a6992359dc46aa52e244542aa62a937fc9f6c3e33dd09369aafb2d422c8 not found: ID does not exist"
Sep 17 03:33:58.614845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.614784    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5" podStartSLOduration=1.614773063 podStartE2EDuration="1.614773063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:33:58.61231147 +0000 UTC m=+889.215407490" watchObservedRunningTime="2026-09-17 03:33:58.614773063 +0000 UTC m=+889.217869058"
Sep 17 03:33:58.630649 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.630612    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct" podStartSLOduration=1.6306009590000001 podStartE2EDuration="1.630600959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:33:58.629430359 +0000 UTC m=+889.232526367" watchObservedRunningTime="2026-09-17 03:33:58.630600959 +0000 UTC m=+889.233696963"
Sep 17 03:33:58.645700 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.645680    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"]
Sep 17 03:33:58.651299 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:58.651272    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3b91ce2e-openshift-default-7dd8f54488-8lflq"]
Sep 17 03:33:59.916182 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:33:59.916150    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af8660e3-04cf-40e1-9b0f-b5d90ed4c271" path="/var/lib/kubelet/pods/af8660e3-04cf-40e1-9b0f-b5d90ed4c271/volumes"
Sep 17 03:34:03.253498 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:03.253413    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f6a774-zmvc2"]
Sep 17 03:34:03.253890 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:03.253875    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af8660e3-04cf-40e1-9b0f-b5d90ed4c271" containerName="istio-proxy"
Sep 17 03:34:03.253943 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:03.253892    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="af8660e3-04cf-40e1-9b0f-b5d90ed4c271" containerName="istio-proxy"
Sep 17 03:34:03.253985 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:03.253975    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="af8660e3-04cf-40e1-9b0f-b5d90ed4c271" containerName="istio-proxy"
Sep 17 03:34:03.258482 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:03.258461    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f6a774-zmvc2"
Sep 17 03:34:03.260606 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:03.260588    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:34:03.264707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:03.264673    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f6a774-zmvc2"]
Sep 17 03:34:03.424365 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:03.424335    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1535af0-0ed6-453f-a5ad-9932746b3a7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-f6a774-zmvc2\" (UID: \"b1535af0-0ed6-453f-a5ad-9932746b3a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f6a774-zmvc2"
Sep 17 03:34:03.525039 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:03.524964    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1535af0-0ed6-453f-a5ad-9932746b3a7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-f6a774-zmvc2\" (UID: \"b1535af0-0ed6-453f-a5ad-9932746b3a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f6a774-zmvc2"
Sep 17 03:34:03.525781 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:03.525758    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1535af0-0ed6-453f-a5ad-9932746b3a7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-f6a774-zmvc2\" (UID: \"b1535af0-0ed6-453f-a5ad-9932746b3a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f6a774-zmvc2"
Sep 17 03:34:03.572743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:03.572717    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f6a774-zmvc2"
Sep 17 03:34:03.854089 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:03.853396    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f6a774-zmvc2"]
Sep 17 03:34:04.631900 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:04.631864    2557 generic.go:356] "Generic (PLEG): container finished" podID="b1535af0-0ed6-453f-a5ad-9932746b3a7c" containerID="dd91b0ad824d28f76a92b5c58901f3a913cd73d563f64b3adcd4c4de792c4adb" exitCode=0
Sep 17 03:34:04.632268 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:04.631942    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f6a774-zmvc2" event={"ID":"b1535af0-0ed6-453f-a5ad-9932746b3a7c","Type":"ContainerDied","Data":"dd91b0ad824d28f76a92b5c58901f3a913cd73d563f64b3adcd4c4de792c4adb"}
Sep 17 03:34:04.632268 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:04.631996    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f6a774-zmvc2" event={"ID":"b1535af0-0ed6-453f-a5ad-9932746b3a7c","Type":"ContainerStarted","Data":"9519b72f0ce2350700a0de764e7829580278f679b0c967ff261a8e09a0028ecb"}
Sep 17 03:34:05.776538 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:05.776514    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f6a774-zmvc2"
Sep 17 03:34:05.946635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:05.946575    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1535af0-0ed6-453f-a5ad-9932746b3a7c-maas-api-service-ca\") pod \"b1535af0-0ed6-453f-a5ad-9932746b3a7c\" (UID: \"b1535af0-0ed6-453f-a5ad-9932746b3a7c\") "
Sep 17 03:34:05.946894 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:05.946871    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1535af0-0ed6-453f-a5ad-9932746b3a7c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1535af0-0ed6-453f-a5ad-9932746b3a7c" (UID: "b1535af0-0ed6-453f-a5ad-9932746b3a7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:34:06.047912 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:06.047882    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1535af0-0ed6-453f-a5ad-9932746b3a7c-maas-api-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:06.644965 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:06.644938    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f6a774-zmvc2"
Sep 17 03:34:06.645122 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:06.644937    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f6a774-zmvc2" event={"ID":"b1535af0-0ed6-453f-a5ad-9932746b3a7c","Type":"ContainerDied","Data":"9519b72f0ce2350700a0de764e7829580278f679b0c967ff261a8e09a0028ecb"}
Sep 17 03:34:06.645122 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:06.645066    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9519b72f0ce2350700a0de764e7829580278f679b0c967ff261a8e09a0028ecb"
Sep 17 03:34:09.604926 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:09.604896    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"
Sep 17 03:34:09.605419 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:09.604989    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"
Sep 17 03:34:09.895352 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:09.895326    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:34:09.895521 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:09.895359    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:34:14.477955 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:14.477916    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd"]
Sep 17 03:34:14.478634 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:14.478612    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af8660e3-04cf-40e1-9b0f-b5d90ed4c271" containerName="istio-proxy"
Sep 17 03:34:14.478729 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:14.478636    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="af8660e3-04cf-40e1-9b0f-b5d90ed4c271" containerName="istio-proxy"
Sep 17 03:34:14.478729 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:14.478658    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1535af0-0ed6-453f-a5ad-9932746b3a7c" containerName="revoke-keys"
Sep 17 03:34:14.478729 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:14.478666    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1535af0-0ed6-453f-a5ad-9932746b3a7c" containerName="revoke-keys"
Sep 17 03:34:14.478877 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:14.478782    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1535af0-0ed6-453f-a5ad-9932746b3a7c" containerName="revoke-keys"
Sep 17 03:34:14.478877 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:14.478795    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="af8660e3-04cf-40e1-9b0f-b5d90ed4c271" containerName="istio-proxy"
Sep 17 03:34:14.483595 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:14.483574    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd"
Sep 17 03:34:14.486030 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:14.486004    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:34:14.488903 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:14.488800    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd"]
Sep 17 03:34:14.522904 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:14.522874    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d20e21-7637-482b-b48f-062bff1c1fed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd\" (UID: \"88d20e21-7637-482b-b48f-062bff1c1fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd"
Sep 17 03:34:14.624823 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:14.624791    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d20e21-7637-482b-b48f-062bff1c1fed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd\" (UID: \"88d20e21-7637-482b-b48f-062bff1c1fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd"
Sep 17 03:34:14.625422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:14.625403    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d20e21-7637-482b-b48f-062bff1c1fed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd\" (UID: \"88d20e21-7637-482b-b48f-062bff1c1fed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd"
Sep 17 03:34:14.795459 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:14.795395    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd"
Sep 17 03:34:15.002805 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.002721    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"]
Sep 17 03:34:15.003640 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.003123    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms" podUID="2b806d1d-5bcc-4607-94c1-4a8ecff47c50" containerName="payload-pre-processing" containerID="cri-o://e5699918e0b08a568b82acc8e2787dde5c3dab24bebc4cecf2b06655d644409d" gracePeriod=30
Sep 17 03:34:15.069307 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.069278    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd"]
Sep 17 03:34:15.077521 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:34:15.077476    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d20e21_7637_482b_b48f_062bff1c1fed.slice/crio-8b2ea358e80a7c0fb9c50cf28301bcf598b152dc50b0f3e1eeaab6e08ebb10bf WatchSource:0}: Error finding container 8b2ea358e80a7c0fb9c50cf28301bcf598b152dc50b0f3e1eeaab6e08ebb10bf: Status 404 returned error can't find the container with id 8b2ea358e80a7c0fb9c50cf28301bcf598b152dc50b0f3e1eeaab6e08ebb10bf
Sep 17 03:34:15.246369 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.246344    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"
Sep 17 03:34:15.333323 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.333251    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkbxm\" (UniqueName: \"kubernetes.io/projected/2b806d1d-5bcc-4607-94c1-4a8ecff47c50-kube-api-access-vkbxm\") pod \"2b806d1d-5bcc-4607-94c1-4a8ecff47c50\" (UID: \"2b806d1d-5bcc-4607-94c1-4a8ecff47c50\") "
Sep 17 03:34:15.333323 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.333319    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b806d1d-5bcc-4607-94c1-4a8ecff47c50-plugins-config-volume\") pod \"2b806d1d-5bcc-4607-94c1-4a8ecff47c50\" (UID: \"2b806d1d-5bcc-4607-94c1-4a8ecff47c50\") "
Sep 17 03:34:15.333882 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.333853    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b806d1d-5bcc-4607-94c1-4a8ecff47c50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b806d1d-5bcc-4607-94c1-4a8ecff47c50" (UID: "2b806d1d-5bcc-4607-94c1-4a8ecff47c50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:34:15.335680 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.335606    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b806d1d-5bcc-4607-94c1-4a8ecff47c50-kube-api-access-vkbxm" (OuterVolumeSpecName: "kube-api-access-vkbxm") pod "2b806d1d-5bcc-4607-94c1-4a8ecff47c50" (UID: "2b806d1d-5bcc-4607-94c1-4a8ecff47c50"). InnerVolumeSpecName "kube-api-access-vkbxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:34:15.338831 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.338802    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f6a774-zmvc2"]
Sep 17 03:34:15.341035 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.341011    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f6a774-zmvc2"]
Sep 17 03:34:15.434541 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.434521    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkbxm\" (UniqueName: \"kubernetes.io/projected/2b806d1d-5bcc-4607-94c1-4a8ecff47c50-kube-api-access-vkbxm\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:15.434541 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.434542    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b806d1d-5bcc-4607-94c1-4a8ecff47c50-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:15.699428 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.699393    2557 generic.go:356] "Generic (PLEG): container finished" podID="2b806d1d-5bcc-4607-94c1-4a8ecff47c50" containerID="e5699918e0b08a568b82acc8e2787dde5c3dab24bebc4cecf2b06655d644409d" exitCode=0
Sep 17 03:34:15.699873 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.699433    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"
Sep 17 03:34:15.699873 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.699485    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms" event={"ID":"2b806d1d-5bcc-4607-94c1-4a8ecff47c50","Type":"ContainerDied","Data":"e5699918e0b08a568b82acc8e2787dde5c3dab24bebc4cecf2b06655d644409d"}
Sep 17 03:34:15.699873 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.699548    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms" event={"ID":"2b806d1d-5bcc-4607-94c1-4a8ecff47c50","Type":"ContainerDied","Data":"da738ce91f038f18823b0437ae714a8f5fa4b2fe78209a234a181fce549669b0"}
Sep 17 03:34:15.699873 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.699586    2557 scope.go:117] "RemoveContainer" containerID="e5699918e0b08a568b82acc8e2787dde5c3dab24bebc4cecf2b06655d644409d"
Sep 17 03:34:15.701270 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.701246    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd" event={"ID":"88d20e21-7637-482b-b48f-062bff1c1fed","Type":"ContainerStarted","Data":"75a7846647f8d977b96d153160513a0ad0f123a1b1f456110928480cd29d290e"}
Sep 17 03:34:15.701387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.701292    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd" event={"ID":"88d20e21-7637-482b-b48f-062bff1c1fed","Type":"ContainerStarted","Data":"8b2ea358e80a7c0fb9c50cf28301bcf598b152dc50b0f3e1eeaab6e08ebb10bf"}
Sep 17 03:34:15.708628 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.708611    2557 scope.go:117] "RemoveContainer" containerID="e5699918e0b08a568b82acc8e2787dde5c3dab24bebc4cecf2b06655d644409d"
Sep 17 03:34:15.708880 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:34:15.708861    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5699918e0b08a568b82acc8e2787dde5c3dab24bebc4cecf2b06655d644409d\": container with ID starting with e5699918e0b08a568b82acc8e2787dde5c3dab24bebc4cecf2b06655d644409d not found: ID does not exist" containerID="e5699918e0b08a568b82acc8e2787dde5c3dab24bebc4cecf2b06655d644409d"
Sep 17 03:34:15.708942 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.708891    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5699918e0b08a568b82acc8e2787dde5c3dab24bebc4cecf2b06655d644409d"} err="failed to get container status \"e5699918e0b08a568b82acc8e2787dde5c3dab24bebc4cecf2b06655d644409d\": rpc error: code = NotFound desc = could not find container \"e5699918e0b08a568b82acc8e2787dde5c3dab24bebc4cecf2b06655d644409d\": container with ID starting with e5699918e0b08a568b82acc8e2787dde5c3dab24bebc4cecf2b06655d644409d not found: ID does not exist"
Sep 17 03:34:15.720456 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.720420    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd" podStartSLOduration=1.7204066199999999 podStartE2EDuration="1.72040662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:34:15.717901998 +0000 UTC m=+906.320998004" watchObservedRunningTime="2026-09-17 03:34:15.72040662 +0000 UTC m=+906.323502676"
Sep 17 03:34:15.731408 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.731379    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"]
Sep 17 03:34:15.737220 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.737183    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f6a774-86f6968b6b-fjfms"]
Sep 17 03:34:15.915858 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.915833    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b806d1d-5bcc-4607-94c1-4a8ecff47c50" path="/var/lib/kubelet/pods/2b806d1d-5bcc-4607-94c1-4a8ecff47c50/volumes"
Sep 17 03:34:15.916203 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:15.916178    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1535af0-0ed6-453f-a5ad-9932746b3a7c" path="/var/lib/kubelet/pods/b1535af0-0ed6-453f-a5ad-9932746b3a7c/volumes"
Sep 17 03:34:16.709758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:16.709724    2557 generic.go:356] "Generic (PLEG): container finished" podID="88d20e21-7637-482b-b48f-062bff1c1fed" containerID="75a7846647f8d977b96d153160513a0ad0f123a1b1f456110928480cd29d290e" exitCode=0
Sep 17 03:34:16.710143 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:16.709780    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd" event={"ID":"88d20e21-7637-482b-b48f-062bff1c1fed","Type":"ContainerDied","Data":"75a7846647f8d977b96d153160513a0ad0f123a1b1f456110928480cd29d290e"}
Sep 17 03:34:17.843840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:17.843818    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd"
Sep 17 03:34:17.959973 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:17.959940    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d20e21-7637-482b-b48f-062bff1c1fed-maas-api-service-ca\") pod \"88d20e21-7637-482b-b48f-062bff1c1fed\" (UID: \"88d20e21-7637-482b-b48f-062bff1c1fed\") "
Sep 17 03:34:17.960386 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:17.960362    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d20e21-7637-482b-b48f-062bff1c1fed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88d20e21-7637-482b-b48f-062bff1c1fed" (UID: "88d20e21-7637-482b-b48f-062bff1c1fed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:34:18.061322 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:18.061250    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d20e21-7637-482b-b48f-062bff1c1fed-maas-api-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:18.723416 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:18.723374    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd" event={"ID":"88d20e21-7637-482b-b48f-062bff1c1fed","Type":"ContainerDied","Data":"8b2ea358e80a7c0fb9c50cf28301bcf598b152dc50b0f3e1eeaab6e08ebb10bf"}
Sep 17 03:34:18.723612 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:18.723439    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b2ea358e80a7c0fb9c50cf28301bcf598b152dc50b0f3e1eeaab6e08ebb10bf"
Sep 17 03:34:18.723612 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:18.723394    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd"
Sep 17 03:34:25.119260 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.119229    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"]
Sep 17 03:34:25.119763 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.119521    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct" podUID="625158c3-9351-4f7d-9e26-cdbde0e0465c" containerName="payload-processing" containerID="cri-o://a73dd908f6c0a5288aeb53103268aa5f38ebdd0de805f2402cec629230ba070b" gracePeriod=30
Sep 17 03:34:25.131543 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.130805    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"]
Sep 17 03:34:25.131543 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.131115    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5" podUID="a1896049-c33c-402a-977e-9944c91a10b0" containerName="payload-pre-processing" containerID="cri-o://dd6aa57d41257285c2a00519c86b08c7652462880c778c3c40bd69ef0109b33a" gracePeriod=30
Sep 17 03:34:25.294637 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.294605    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd"]
Sep 17 03:34:25.299787 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.299579    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-acf311cd-cbhsd"]
Sep 17 03:34:25.422490 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.422045    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"
Sep 17 03:34:25.462699 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.462676    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"
Sep 17 03:34:25.529750 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.529719    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1896049-c33c-402a-977e-9944c91a10b0-plugins-config-volume\") pod \"a1896049-c33c-402a-977e-9944c91a10b0\" (UID: \"a1896049-c33c-402a-977e-9944c91a10b0\") "
Sep 17 03:34:25.530002 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.529980    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnm6z\" (UniqueName: \"kubernetes.io/projected/a1896049-c33c-402a-977e-9944c91a10b0-kube-api-access-wnm6z\") pod \"a1896049-c33c-402a-977e-9944c91a10b0\" (UID: \"a1896049-c33c-402a-977e-9944c91a10b0\") "
Sep 17 03:34:25.530235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.530211    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625158c3-9351-4f7d-9e26-cdbde0e0465c-plugins-config-volume\") pod \"625158c3-9351-4f7d-9e26-cdbde0e0465c\" (UID: \"625158c3-9351-4f7d-9e26-cdbde0e0465c\") "
Sep 17 03:34:25.530415 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.530397    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdzbj\" (UniqueName: \"kubernetes.io/projected/625158c3-9351-4f7d-9e26-cdbde0e0465c-kube-api-access-vdzbj\") pod \"625158c3-9351-4f7d-9e26-cdbde0e0465c\" (UID: \"625158c3-9351-4f7d-9e26-cdbde0e0465c\") "
Sep 17 03:34:25.533826 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.533618    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625158c3-9351-4f7d-9e26-cdbde0e0465c-kube-api-access-vdzbj" (OuterVolumeSpecName: "kube-api-access-vdzbj") pod "625158c3-9351-4f7d-9e26-cdbde0e0465c" (UID: "625158c3-9351-4f7d-9e26-cdbde0e0465c"). InnerVolumeSpecName "kube-api-access-vdzbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:34:25.534371 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.534087    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1896049-c33c-402a-977e-9944c91a10b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1896049-c33c-402a-977e-9944c91a10b0" (UID: "a1896049-c33c-402a-977e-9944c91a10b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:34:25.536808 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.536627    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1896049-c33c-402a-977e-9944c91a10b0-kube-api-access-wnm6z" (OuterVolumeSpecName: "kube-api-access-wnm6z") pod "a1896049-c33c-402a-977e-9944c91a10b0" (UID: "a1896049-c33c-402a-977e-9944c91a10b0"). InnerVolumeSpecName "kube-api-access-wnm6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:34:25.537092 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.537061    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625158c3-9351-4f7d-9e26-cdbde0e0465c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "625158c3-9351-4f7d-9e26-cdbde0e0465c" (UID: "625158c3-9351-4f7d-9e26-cdbde0e0465c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:34:25.631580 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.631538    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1896049-c33c-402a-977e-9944c91a10b0-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:25.631580 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.631571    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnm6z\" (UniqueName: \"kubernetes.io/projected/a1896049-c33c-402a-977e-9944c91a10b0-kube-api-access-wnm6z\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:25.631580 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.631584    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/625158c3-9351-4f7d-9e26-cdbde0e0465c-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:25.631813 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.631597    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdzbj\" (UniqueName: \"kubernetes.io/projected/625158c3-9351-4f7d-9e26-cdbde0e0465c-kube-api-access-vdzbj\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:25.764997 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.764921    2557 generic.go:356] "Generic (PLEG): container finished" podID="a1896049-c33c-402a-977e-9944c91a10b0" containerID="dd6aa57d41257285c2a00519c86b08c7652462880c778c3c40bd69ef0109b33a" exitCode=0
Sep 17 03:34:25.764997 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.764983    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"
Sep 17 03:34:25.765222 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.765001    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5" event={"ID":"a1896049-c33c-402a-977e-9944c91a10b0","Type":"ContainerDied","Data":"dd6aa57d41257285c2a00519c86b08c7652462880c778c3c40bd69ef0109b33a"}
Sep 17 03:34:25.765222 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.765077    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5" event={"ID":"a1896049-c33c-402a-977e-9944c91a10b0","Type":"ContainerDied","Data":"e30be4101aca9448749e09f6e1d3bb875942b51ddc6ab3d9fd1a51311ff20793"}
Sep 17 03:34:25.765222 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.765107    2557 scope.go:117] "RemoveContainer" containerID="dd6aa57d41257285c2a00519c86b08c7652462880c778c3c40bd69ef0109b33a"
Sep 17 03:34:25.766548 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.766521    2557 generic.go:356] "Generic (PLEG): container finished" podID="625158c3-9351-4f7d-9e26-cdbde0e0465c" containerID="a73dd908f6c0a5288aeb53103268aa5f38ebdd0de805f2402cec629230ba070b" exitCode=0
Sep 17 03:34:25.766682 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.766583    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"
Sep 17 03:34:25.766682 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.766577    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct" event={"ID":"625158c3-9351-4f7d-9e26-cdbde0e0465c","Type":"ContainerDied","Data":"a73dd908f6c0a5288aeb53103268aa5f38ebdd0de805f2402cec629230ba070b"}
Sep 17 03:34:25.766796 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.766722    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct" event={"ID":"625158c3-9351-4f7d-9e26-cdbde0e0465c","Type":"ContainerDied","Data":"98ce2503ba60a370af450f98b76792acd7637b66d76792a01eb8144cf25d9813"}
Sep 17 03:34:25.776108 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.776091    2557 scope.go:117] "RemoveContainer" containerID="dd6aa57d41257285c2a00519c86b08c7652462880c778c3c40bd69ef0109b33a"
Sep 17 03:34:25.776482 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:34:25.776457    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6aa57d41257285c2a00519c86b08c7652462880c778c3c40bd69ef0109b33a\": container with ID starting with dd6aa57d41257285c2a00519c86b08c7652462880c778c3c40bd69ef0109b33a not found: ID does not exist" containerID="dd6aa57d41257285c2a00519c86b08c7652462880c778c3c40bd69ef0109b33a"
Sep 17 03:34:25.776586 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.776487    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6aa57d41257285c2a00519c86b08c7652462880c778c3c40bd69ef0109b33a"} err="failed to get container status \"dd6aa57d41257285c2a00519c86b08c7652462880c778c3c40bd69ef0109b33a\": rpc error: code = NotFound desc = could not find container \"dd6aa57d41257285c2a00519c86b08c7652462880c778c3c40bd69ef0109b33a\": container with ID starting with dd6aa57d41257285c2a00519c86b08c7652462880c778c3c40bd69ef0109b33a not found: ID does not exist"
Sep 17 03:34:25.776586 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.776503    2557 scope.go:117] "RemoveContainer" containerID="a73dd908f6c0a5288aeb53103268aa5f38ebdd0de805f2402cec629230ba070b"
Sep 17 03:34:25.785234 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.785186    2557 scope.go:117] "RemoveContainer" containerID="a73dd908f6c0a5288aeb53103268aa5f38ebdd0de805f2402cec629230ba070b"
Sep 17 03:34:25.785475 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:34:25.785457    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73dd908f6c0a5288aeb53103268aa5f38ebdd0de805f2402cec629230ba070b\": container with ID starting with a73dd908f6c0a5288aeb53103268aa5f38ebdd0de805f2402cec629230ba070b not found: ID does not exist" containerID="a73dd908f6c0a5288aeb53103268aa5f38ebdd0de805f2402cec629230ba070b"
Sep 17 03:34:25.785526 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.785480    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73dd908f6c0a5288aeb53103268aa5f38ebdd0de805f2402cec629230ba070b"} err="failed to get container status \"a73dd908f6c0a5288aeb53103268aa5f38ebdd0de805f2402cec629230ba070b\": rpc error: code = NotFound desc = could not find container \"a73dd908f6c0a5288aeb53103268aa5f38ebdd0de805f2402cec629230ba070b\": container with ID starting with a73dd908f6c0a5288aeb53103268aa5f38ebdd0de805f2402cec629230ba070b not found: ID does not exist"
Sep 17 03:34:25.797107 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.797084    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"]
Sep 17 03:34:25.799230 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.799208    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-acf311cd-cd998f5c4-csbf5"]
Sep 17 03:34:25.808715 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.808694    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"]
Sep 17 03:34:25.816825 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.816806    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-acf311cd-6db5c658bf-5d6ct"]
Sep 17 03:34:25.915758 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.915730    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="625158c3-9351-4f7d-9e26-cdbde0e0465c" path="/var/lib/kubelet/pods/625158c3-9351-4f7d-9e26-cdbde0e0465c/volumes"
Sep 17 03:34:25.916101 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.916088    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d20e21-7637-482b-b48f-062bff1c1fed" path="/var/lib/kubelet/pods/88d20e21-7637-482b-b48f-062bff1c1fed/volumes"
Sep 17 03:34:25.916449 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:25.916427    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1896049-c33c-402a-977e-9944c91a10b0" path="/var/lib/kubelet/pods/a1896049-c33c-402a-977e-9944c91a10b0/volumes"
Sep 17 03:34:28.070369 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.070335    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"]
Sep 17 03:34:28.070858 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.070553    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4" podUID="7025aadb-3ef1-42af-b67c-b4004b19dffe" containerName="istio-proxy" containerID="cri-o://506e01e1e304c1da0769b8959d0ab089dfe9fa0bc0c08ad7da1b62e08923f6ae" gracePeriod=30
Sep 17 03:34:28.796736 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.796708    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"]
Sep 17 03:34:28.797173 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.797161    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b806d1d-5bcc-4607-94c1-4a8ecff47c50" containerName="payload-pre-processing"
Sep 17 03:34:28.797243 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.797175    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b806d1d-5bcc-4607-94c1-4a8ecff47c50" containerName="payload-pre-processing"
Sep 17 03:34:28.797243 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.797205    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1896049-c33c-402a-977e-9944c91a10b0" containerName="payload-pre-processing"
Sep 17 03:34:28.797243 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.797211    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1896049-c33c-402a-977e-9944c91a10b0" containerName="payload-pre-processing"
Sep 17 03:34:28.797243 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.797235    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="625158c3-9351-4f7d-9e26-cdbde0e0465c" containerName="payload-processing"
Sep 17 03:34:28.797243 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.797244    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="625158c3-9351-4f7d-9e26-cdbde0e0465c" containerName="payload-processing"
Sep 17 03:34:28.797396 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.797252    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d20e21-7637-482b-b48f-062bff1c1fed" containerName="revoke-keys"
Sep 17 03:34:28.797396 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.797257    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d20e21-7637-482b-b48f-062bff1c1fed" containerName="revoke-keys"
Sep 17 03:34:28.797396 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.797329    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d20e21-7637-482b-b48f-062bff1c1fed" containerName="revoke-keys"
Sep 17 03:34:28.797396 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.797340    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1896049-c33c-402a-977e-9944c91a10b0" containerName="payload-pre-processing"
Sep 17 03:34:28.797396 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.797348    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b806d1d-5bcc-4607-94c1-4a8ecff47c50" containerName="payload-pre-processing"
Sep 17 03:34:28.797396 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.797358    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="625158c3-9351-4f7d-9e26-cdbde0e0465c" containerName="payload-processing"
Sep 17 03:34:28.802002 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.801974    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.804310 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.804286    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-5062bf-openshift-default-dockercfg-tddpl\""
Sep 17 03:34:28.817788 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.817762    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"]
Sep 17 03:34:28.861481 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.861459    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ba5071-edfb-4133-836b-531abf633199-istio-podinfo\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.861599 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.861500    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-istio-envoy\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.861599 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.861526    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dcrt\" (UniqueName: \"kubernetes.io/projected/72ba5071-edfb-4133-836b-531abf633199-kube-api-access-6dcrt\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.861715 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.861595    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ba5071-edfb-4133-836b-531abf633199-istiod-ca-cert\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.861715 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.861639    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ba5071-edfb-4133-836b-531abf633199-istio-token\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.861715 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.861663    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-istio-data\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.861846 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.861724    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-credential-socket\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.861846 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.861745    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-workload-certs\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.861846 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.861815    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-workload-socket\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.962755 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.962718    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-credential-socket\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.962755 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.962759    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-workload-certs\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.963005 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.962792    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-workload-socket\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.963005 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.962835    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ba5071-edfb-4133-836b-531abf633199-istio-podinfo\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.963005 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.962881    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-istio-envoy\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.963005 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.962919    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dcrt\" (UniqueName: \"kubernetes.io/projected/72ba5071-edfb-4133-836b-531abf633199-kube-api-access-6dcrt\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.963005 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.962978    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ba5071-edfb-4133-836b-531abf633199-istiod-ca-cert\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.963321 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.963024    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ba5071-edfb-4133-836b-531abf633199-istio-token\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.963321 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.963062    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-istio-data\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.963321 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.963183    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-workload-certs\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.963321 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.963300    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-workload-socket\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.963542 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.963469    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-istio-data\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.963761 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.963737    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-credential-socket\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.963912 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.963778    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ba5071-edfb-4133-836b-531abf633199-istiod-ca-cert\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.965452 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.965434    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ba5071-edfb-4133-836b-531abf633199-istio-podinfo\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.965670 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.965653    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-istio-envoy\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.974382 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.974363    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ba5071-edfb-4133-836b-531abf633199-istio-token\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:28.975116 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:28.975091    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dcrt\" (UniqueName: \"kubernetes.io/projected/72ba5071-edfb-4133-836b-531abf633199-kube-api-access-6dcrt\") pod \"e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") " pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:29.113268 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:29.113179    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:29.400414 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:29.400379    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"]
Sep 17 03:34:29.401821 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:34:29.401792    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ba5071_edfb_4133_836b_531abf633199.slice/crio-83074b9503da20f8c6e9e569692dd5f017ef71bbe6e12545ad8db0021ac860c7 WatchSource:0}: Error finding container 83074b9503da20f8c6e9e569692dd5f017ef71bbe6e12545ad8db0021ac860c7: Status 404 returned error can't find the container with id 83074b9503da20f8c6e9e569692dd5f017ef71bbe6e12545ad8db0021ac860c7
Sep 17 03:34:29.404241 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:29.404169    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:34:29.404341 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:29.404271    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:34:29.404341 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:29.404303    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:34:29.795439 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:29.795355    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h" event={"ID":"72ba5071-edfb-4133-836b-531abf633199","Type":"ContainerStarted","Data":"bccace71ee29b4672ff9ac4a7c6609b31a3edfff53eadec8aa5e767bada85427"}
Sep 17 03:34:29.795439 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:29.795407    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h" event={"ID":"72ba5071-edfb-4133-836b-531abf633199","Type":"ContainerStarted","Data":"83074b9503da20f8c6e9e569692dd5f017ef71bbe6e12545ad8db0021ac860c7"}
Sep 17 03:34:29.818149 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:29.818091    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h" podStartSLOduration=1.818072846 podStartE2EDuration="1.818072846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:34:29.81731757 +0000 UTC m=+920.420413585" watchObservedRunningTime="2026-09-17 03:34:29.818072846 +0000 UTC m=+920.421168850"
Sep 17 03:34:30.113399 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:30.113320    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:30.119827 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:30.119803    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:30.801287 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:30.801250    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:30.802341 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:30.802320    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:33.338061 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.337997    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:34:33.406924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.406896    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-token\") pod \"7025aadb-3ef1-42af-b67c-b4004b19dffe\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") "
Sep 17 03:34:33.407074 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.406938    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-data\") pod \"7025aadb-3ef1-42af-b67c-b4004b19dffe\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") "
Sep 17 03:34:33.407074 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.406970    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-envoy\") pod \"7025aadb-3ef1-42af-b67c-b4004b19dffe\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") "
Sep 17 03:34:33.407074 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407006    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7025aadb-3ef1-42af-b67c-b4004b19dffe-istiod-ca-cert\") pod \"7025aadb-3ef1-42af-b67c-b4004b19dffe\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") "
Sep 17 03:34:33.407074 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407034    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2tpx\" (UniqueName: \"kubernetes.io/projected/7025aadb-3ef1-42af-b67c-b4004b19dffe-kube-api-access-f2tpx\") pod \"7025aadb-3ef1-42af-b67c-b4004b19dffe\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") "
Sep 17 03:34:33.407074 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407067    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-workload-certs\") pod \"7025aadb-3ef1-42af-b67c-b4004b19dffe\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") "
Sep 17 03:34:33.407421 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407118    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-credential-socket\") pod \"7025aadb-3ef1-42af-b67c-b4004b19dffe\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") "
Sep 17 03:34:33.407421 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407179    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-podinfo\") pod \"7025aadb-3ef1-42af-b67c-b4004b19dffe\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") "
Sep 17 03:34:33.407421 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407266    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-workload-socket\") pod \"7025aadb-3ef1-42af-b67c-b4004b19dffe\" (UID: \"7025aadb-3ef1-42af-b67c-b4004b19dffe\") "
Sep 17 03:34:33.407421 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407340    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-data" (OuterVolumeSpecName: "istio-data") pod "7025aadb-3ef1-42af-b67c-b4004b19dffe" (UID: "7025aadb-3ef1-42af-b67c-b4004b19dffe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:34:33.407639 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407498    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7025aadb-3ef1-42af-b67c-b4004b19dffe" (UID: "7025aadb-3ef1-42af-b67c-b4004b19dffe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:34:33.407639 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407519    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7025aadb-3ef1-42af-b67c-b4004b19dffe" (UID: "7025aadb-3ef1-42af-b67c-b4004b19dffe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:34:33.407639 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407529    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7025aadb-3ef1-42af-b67c-b4004b19dffe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7025aadb-3ef1-42af-b67c-b4004b19dffe" (UID: "7025aadb-3ef1-42af-b67c-b4004b19dffe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:34:33.407794 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407706    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7025aadb-3ef1-42af-b67c-b4004b19dffe" (UID: "7025aadb-3ef1-42af-b67c-b4004b19dffe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:34:33.407854 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407796    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:33.407854 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407813    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7025aadb-3ef1-42af-b67c-b4004b19dffe-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:33.407854 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407826    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:33.407854 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407838    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:33.407854 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.407850    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:33.409282 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.409254    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7025aadb-3ef1-42af-b67c-b4004b19dffe" (UID: "7025aadb-3ef1-42af-b67c-b4004b19dffe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:34:33.409421 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.409302    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-token" (OuterVolumeSpecName: "istio-token") pod "7025aadb-3ef1-42af-b67c-b4004b19dffe" (UID: "7025aadb-3ef1-42af-b67c-b4004b19dffe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:34:33.409637 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.409617    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7025aadb-3ef1-42af-b67c-b4004b19dffe" (UID: "7025aadb-3ef1-42af-b67c-b4004b19dffe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:34:33.409743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.409674    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7025aadb-3ef1-42af-b67c-b4004b19dffe-kube-api-access-f2tpx" (OuterVolumeSpecName: "kube-api-access-f2tpx") pod "7025aadb-3ef1-42af-b67c-b4004b19dffe" (UID: "7025aadb-3ef1-42af-b67c-b4004b19dffe"). InnerVolumeSpecName "kube-api-access-f2tpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:34:33.509285 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.509266    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:33.509285 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.509285    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:33.509426 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.509295    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7025aadb-3ef1-42af-b67c-b4004b19dffe-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:33.509426 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.509303    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2tpx\" (UniqueName: \"kubernetes.io/projected/7025aadb-3ef1-42af-b67c-b4004b19dffe-kube-api-access-f2tpx\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:34:33.821259 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.821226    2557 generic.go:356] "Generic (PLEG): container finished" podID="7025aadb-3ef1-42af-b67c-b4004b19dffe" containerID="506e01e1e304c1da0769b8959d0ab089dfe9fa0bc0c08ad7da1b62e08923f6ae" exitCode=0
Sep 17 03:34:33.821426 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.821327    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4" event={"ID":"7025aadb-3ef1-42af-b67c-b4004b19dffe","Type":"ContainerDied","Data":"506e01e1e304c1da0769b8959d0ab089dfe9fa0bc0c08ad7da1b62e08923f6ae"}
Sep 17 03:34:33.821426 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.821352    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"
Sep 17 03:34:33.821426 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.821378    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4" event={"ID":"7025aadb-3ef1-42af-b67c-b4004b19dffe","Type":"ContainerDied","Data":"2a7fd660b32a2c5fde892e0df4223db373b2f030e133832a8508f85ec9ff820e"}
Sep 17 03:34:33.821426 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.821406    2557 scope.go:117] "RemoveContainer" containerID="506e01e1e304c1da0769b8959d0ab089dfe9fa0bc0c08ad7da1b62e08923f6ae"
Sep 17 03:34:33.831940 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.831725    2557 scope.go:117] "RemoveContainer" containerID="4b4e44e173b9b688c57c9eb7710e1d057b4f863c5f0bf0852e60bf7c47cf8b3f"
Sep 17 03:34:33.840780 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.840762    2557 scope.go:117] "RemoveContainer" containerID="506e01e1e304c1da0769b8959d0ab089dfe9fa0bc0c08ad7da1b62e08923f6ae"
Sep 17 03:34:33.841050 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:34:33.841034    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506e01e1e304c1da0769b8959d0ab089dfe9fa0bc0c08ad7da1b62e08923f6ae\": container with ID starting with 506e01e1e304c1da0769b8959d0ab089dfe9fa0bc0c08ad7da1b62e08923f6ae not found: ID does not exist" containerID="506e01e1e304c1da0769b8959d0ab089dfe9fa0bc0c08ad7da1b62e08923f6ae"
Sep 17 03:34:33.841131 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.841059    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506e01e1e304c1da0769b8959d0ab089dfe9fa0bc0c08ad7da1b62e08923f6ae"} err="failed to get container status \"506e01e1e304c1da0769b8959d0ab089dfe9fa0bc0c08ad7da1b62e08923f6ae\": rpc error: code = NotFound desc = could not find container \"506e01e1e304c1da0769b8959d0ab089dfe9fa0bc0c08ad7da1b62e08923f6ae\": container with ID starting with 506e01e1e304c1da0769b8959d0ab089dfe9fa0bc0c08ad7da1b62e08923f6ae not found: ID does not exist"
Sep 17 03:34:33.841131 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.841074    2557 scope.go:117] "RemoveContainer" containerID="4b4e44e173b9b688c57c9eb7710e1d057b4f863c5f0bf0852e60bf7c47cf8b3f"
Sep 17 03:34:33.841319 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:34:33.841304    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b4e44e173b9b688c57c9eb7710e1d057b4f863c5f0bf0852e60bf7c47cf8b3f\": container with ID starting with 4b4e44e173b9b688c57c9eb7710e1d057b4f863c5f0bf0852e60bf7c47cf8b3f not found: ID does not exist" containerID="4b4e44e173b9b688c57c9eb7710e1d057b4f863c5f0bf0852e60bf7c47cf8b3f"
Sep 17 03:34:33.841363 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.841323    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b4e44e173b9b688c57c9eb7710e1d057b4f863c5f0bf0852e60bf7c47cf8b3f"} err="failed to get container status \"4b4e44e173b9b688c57c9eb7710e1d057b4f863c5f0bf0852e60bf7c47cf8b3f\": rpc error: code = NotFound desc = could not find container \"4b4e44e173b9b688c57c9eb7710e1d057b4f863c5f0bf0852e60bf7c47cf8b3f\": container with ID starting with 4b4e44e173b9b688c57c9eb7710e1d057b4f863c5f0bf0852e60bf7c47cf8b3f not found: ID does not exist"
Sep 17 03:34:33.846230 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.846207    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"]
Sep 17 03:34:33.851264 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.851242    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-f6a774-openshift-default-7ddfffc5b-rnvx4"]
Sep 17 03:34:33.915636 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:33.915612    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7025aadb-3ef1-42af-b67c-b4004b19dffe" path="/var/lib/kubelet/pods/7025aadb-3ef1-42af-b67c-b4004b19dffe/volumes"
Sep 17 03:34:34.721819 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.721786    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"]
Sep 17 03:34:34.726579 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.725615    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7025aadb-3ef1-42af-b67c-b4004b19dffe" containerName="istio-proxy"
Sep 17 03:34:34.726579 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.725645    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7025aadb-3ef1-42af-b67c-b4004b19dffe" containerName="istio-proxy"
Sep 17 03:34:34.726579 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.725886    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7025aadb-3ef1-42af-b67c-b4004b19dffe" containerName="istio-proxy"
Sep 17 03:34:34.726579 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.725910    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7025aadb-3ef1-42af-b67c-b4004b19dffe" containerName="istio-proxy"
Sep 17 03:34:34.731985 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.731074    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:34.734003 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.733981    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-5062bf-serving-cert\""
Sep 17 03:34:34.761107 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.761083    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"]
Sep 17 03:34:34.821929 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.821905    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h226g\" (UniqueName: \"kubernetes.io/projected/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-kube-api-access-h226g\") pod \"maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:34.822072 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.821965    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-metrics-certs\") pod \"maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:34.822072 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.822046    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-maas-api-tls\") pod \"maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:34.822178 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.822090    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-openshift-service-ca\") pod \"maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:34.923623 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.923593    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-maas-api-tls\") pod \"maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:34.923747 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.923641    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-openshift-service-ca\") pod \"maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:34.923747 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.923717    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h226g\" (UniqueName: \"kubernetes.io/projected/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-kube-api-access-h226g\") pod \"maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:34.923857 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:34:34.923755    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-serving-cert: secret "maas-api-e2e-explicit-5062bf-serving-cert" not found
Sep 17 03:34:34.923857 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.923790    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-metrics-certs\") pod \"maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:34.923857 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:34:34.923833    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-maas-api-tls podName:2b557adf-444c-4a8d-8c2e-0fe3a6faf348 nodeName:}" failed. No retries permitted until 2026-09-17 03:34:35.423812555 +0000 UTC m=+926.026908542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-maas-api-tls") pod "maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj" (UID: "2b557adf-444c-4a8d-8c2e-0fe3a6faf348") : secret "maas-api-e2e-explicit-5062bf-serving-cert" not found
Sep 17 03:34:34.924440 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.924414    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-openshift-service-ca\") pod \"maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:34.926323 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.926299    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-metrics-certs\") pod \"maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:34.935358 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:34.935338    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h226g\" (UniqueName: \"kubernetes.io/projected/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-kube-api-access-h226g\") pod \"maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:35.430014 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.429977    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-maas-api-tls\") pod \"maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:35.432709 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.432681    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-maas-api-tls\") pod \"maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:35.521162 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.521132    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"]
Sep 17 03:34:35.521631 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.521616    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7025aadb-3ef1-42af-b67c-b4004b19dffe" containerName="istio-proxy"
Sep 17 03:34:35.521631 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.521630    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7025aadb-3ef1-42af-b67c-b4004b19dffe" containerName="istio-proxy"
Sep 17 03:34:35.525994 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.525973    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"
Sep 17 03:34:35.528265 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.528244    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-5062bf-dockercfg-lhdsx\""
Sep 17 03:34:35.528358 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.528284    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-5062bf\""
Sep 17 03:34:35.536775 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.536746    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"]
Sep 17 03:34:35.632263 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.632229    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52snl\" (UniqueName: \"kubernetes.io/projected/22c1d400-a6ba-41f2-ab48-9d45c5887ab8-kube-api-access-52snl\") pod \"payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6\" (UID: \"22c1d400-a6ba-41f2-ab48-9d45c5887ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"
Sep 17 03:34:35.632629 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.632607    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c1d400-a6ba-41f2-ab48-9d45c5887ab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6\" (UID: \"22c1d400-a6ba-41f2-ab48-9d45c5887ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"
Sep 17 03:34:35.643705 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.643679    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:35.733956 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.733917    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52snl\" (UniqueName: \"kubernetes.io/projected/22c1d400-a6ba-41f2-ab48-9d45c5887ab8-kube-api-access-52snl\") pod \"payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6\" (UID: \"22c1d400-a6ba-41f2-ab48-9d45c5887ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"
Sep 17 03:34:35.734387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.734023    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c1d400-a6ba-41f2-ab48-9d45c5887ab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6\" (UID: \"22c1d400-a6ba-41f2-ab48-9d45c5887ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"
Sep 17 03:34:35.734878 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.734852    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c1d400-a6ba-41f2-ab48-9d45c5887ab8-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6\" (UID: \"22c1d400-a6ba-41f2-ab48-9d45c5887ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"
Sep 17 03:34:35.743832 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.743808    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52snl\" (UniqueName: \"kubernetes.io/projected/22c1d400-a6ba-41f2-ab48-9d45c5887ab8-kube-api-access-52snl\") pod \"payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6\" (UID: \"22c1d400-a6ba-41f2-ab48-9d45c5887ab8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"
Sep 17 03:34:35.838100 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.838060    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"
Sep 17 03:34:35.928525 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.928499    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"]
Sep 17 03:34:35.930182 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:34:35.930150    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b557adf_444c_4a8d_8c2e_0fe3a6faf348.slice/crio-71db437460157f7997ecab40408989128cb12a9f8b180b7d9a79c8dbc405bccb WatchSource:0}: Error finding container 71db437460157f7997ecab40408989128cb12a9f8b180b7d9a79c8dbc405bccb: Status 404 returned error can't find the container with id 71db437460157f7997ecab40408989128cb12a9f8b180b7d9a79c8dbc405bccb
Sep 17 03:34:35.931629 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:35.931609    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 03:34:36.113051 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:36.112990    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"]
Sep 17 03:34:36.116293 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:34:36.116266    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c1d400_a6ba_41f2_ab48_9d45c5887ab8.slice/crio-4d13fc020d7a88345da30214d23c43371973c5959b75c4952a9a058360fcccac WatchSource:0}: Error finding container 4d13fc020d7a88345da30214d23c43371973c5959b75c4952a9a058360fcccac: Status 404 returned error can't find the container with id 4d13fc020d7a88345da30214d23c43371973c5959b75c4952a9a058360fcccac
Sep 17 03:34:36.843124 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:36.843038    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj" event={"ID":"2b557adf-444c-4a8d-8c2e-0fe3a6faf348","Type":"ContainerStarted","Data":"2dd39243a7dd7ce4220c38a5406d4b5f7d3628eb4cee9140acb58a6c6bfe0ae1"}
Sep 17 03:34:36.843124 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:36.843099    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj" event={"ID":"2b557adf-444c-4a8d-8c2e-0fe3a6faf348","Type":"ContainerStarted","Data":"71db437460157f7997ecab40408989128cb12a9f8b180b7d9a79c8dbc405bccb"}
Sep 17 03:34:36.843646 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:36.843130    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:36.844802 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:36.844778    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6" event={"ID":"22c1d400-a6ba-41f2-ab48-9d45c5887ab8","Type":"ContainerStarted","Data":"fdc14dc54f671380e19569e30c3d043dbbde7af6de6ee767ec90afa0e5a3d840"}
Sep 17 03:34:36.844933 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:36.844822    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6" event={"ID":"22c1d400-a6ba-41f2-ab48-9d45c5887ab8","Type":"ContainerStarted","Data":"4d13fc020d7a88345da30214d23c43371973c5959b75c4952a9a058360fcccac"}
Sep 17 03:34:36.844933 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:36.844859    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"
Sep 17 03:34:36.860145 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:36.860094    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj" podStartSLOduration=2.473798313 podStartE2EDuration="2.860077834s" totalImagesPullingTime="386.279521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:34:36.859741813 +0000 UTC m=+927.462837817" watchObservedRunningTime="2026-09-17 03:34:36.860077834 +0000 UTC m=+927.463173839"
Sep 17 03:34:36.880274 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:36.880235    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6" podStartSLOduration=1.880222193 podStartE2EDuration="1.880222193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:34:36.878385802 +0000 UTC m=+927.481481883" watchObservedRunningTime="2026-09-17 03:34:36.880222193 +0000 UTC m=+927.483318219"
Sep 17 03:34:42.856377 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:42.856347    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:34:47.853317 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:47.853206    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"
Sep 17 03:34:53.943267 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:53.943236    2557 generic.go:356] "Generic (PLEG): container finished" podID="72ba5071-edfb-4133-836b-531abf633199" containerID="bccace71ee29b4672ff9ac4a7c6609b31a3edfff53eadec8aa5e767bada85427" exitCode=0
Sep 17 03:34:53.943676 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:53.943313    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h" event={"ID":"72ba5071-edfb-4133-836b-531abf633199","Type":"ContainerDied","Data":"bccace71ee29b4672ff9ac4a7c6609b31a3edfff53eadec8aa5e767bada85427"}
Sep 17 03:34:53.943676 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:53.943640    2557 scope.go:117] "RemoveContainer" containerID="bccace71ee29b4672ff9ac4a7c6609b31a3edfff53eadec8aa5e767bada85427"
Sep 17 03:34:53.944696 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:53.944666    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:34:53.944760 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:53.944736    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:34:53.944802 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:53.944768    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:34:54.953412 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:54.953380    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h" event={"ID":"72ba5071-edfb-4133-836b-531abf633199","Type":"ContainerStarted","Data":"6214dfb9284e0dc7a9ebde4b39b42ca1a4ed1ddfacdd9a4cfc3db2e0bc156e72"}
Sep 17 03:34:55.114242 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:55.114183    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:34:55.114499 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:55.114473    2557 patch_prober.go:28] interesting pod/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 17 03:34:55.114550 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:55.114530    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h" podUID="72ba5071-edfb-4133-836b-531abf633199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 17 03:34:56.114340 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:56.114309    2557 patch_prober.go:28] interesting pod/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 17 03:34:56.114715 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:56.114368    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h" podUID="72ba5071-edfb-4133-836b-531abf633199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 17 03:34:57.114303 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:57.114268    2557 patch_prober.go:28] interesting pod/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 17 03:34:57.114458 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:57.114326    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h" podUID="72ba5071-edfb-4133-836b-531abf633199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 17 03:34:58.114115 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:58.114086    2557 patch_prober.go:28] interesting pod/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 17 03:34:58.114307 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:58.114138    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h" podUID="72ba5071-edfb-4133-836b-531abf633199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 17 03:34:59.114268 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:59.114234    2557 patch_prober.go:28] interesting pod/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 17 03:34:59.114728 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:34:59.114284    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h" podUID="72ba5071-edfb-4133-836b-531abf633199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 17 03:35:00.137168 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:35:00.137130    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:35:00.137675 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:35:00.137627    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:35:00.140395 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:35:00.140367    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:36:02.819596 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:02.819563    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"]
Sep 17 03:36:02.823982 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:02.823958    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:02.826550 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:02.826525    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cdaa7ee2-openshift-default-dockercfg-sbr5v\""
Sep 17 03:36:02.834920 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:02.834884    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"]
Sep 17 03:36:02.904898 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:02.904868    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-workload-socket\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:02.905032 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:02.904905    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-data\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:02.905032 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:02.904937    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-credential-socket\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:02.905032 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:02.905007    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-workload-certs\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:02.905302 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:02.905068    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhcg5\" (UniqueName: \"kubernetes.io/projected/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-kube-api-access-xhcg5\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:02.905302 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:02.905102    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-token\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:02.905302 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:02.905159    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-podinfo\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:02.905302 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:02.905237    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-envoy\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:02.905302 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:02.905295    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istiod-ca-cert\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.006132 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.006100    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhcg5\" (UniqueName: \"kubernetes.io/projected/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-kube-api-access-xhcg5\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.006346 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.006146    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-token\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.006346 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.006188    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-podinfo\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.006346 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.006229    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-envoy\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.006346 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.006280    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istiod-ca-cert\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.006346 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.006339    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-workload-socket\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.006618 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.006365    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-data\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.006618 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.006403    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-credential-socket\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.006618 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.006455    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-workload-certs\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.006832 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.006782    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-workload-socket\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.006969 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.006835    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-data\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.006969 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.006848    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-workload-certs\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.006969 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.006893    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-credential-socket\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.007413 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.007392    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istiod-ca-cert\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.009099 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.009076    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-envoy\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.009339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.009321    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-podinfo\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.014813 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.014794    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-token\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.014971 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.014953    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhcg5\" (UniqueName: \"kubernetes.io/projected/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-kube-api-access-xhcg5\") pod \"e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") " pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.138775 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.138751    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:03.431685 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.424828    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"]
Sep 17 03:36:03.435449 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.435410    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:36:03.435573 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.435520    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:36:03.435573 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:03.435566    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:36:04.341487 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:04.341456    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj" event={"ID":"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff","Type":"ContainerStarted","Data":"040ba2507e949b1388569f935f5c3e7efac6cec007b841f565e58250bb686369"}
Sep 17 03:36:04.341832 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:04.341505    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj" event={"ID":"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff","Type":"ContainerStarted","Data":"c90252722e9422dc038b3980ed656be3eb43d2eefa3f21a44d922f67ee962f97"}
Sep 17 03:36:04.365095 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:04.365047    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj" podStartSLOduration=2.365031053 podStartE2EDuration="2.365031053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:36:04.364694315 +0000 UTC m=+1014.967790320" watchObservedRunningTime="2026-09-17 03:36:04.365031053 +0000 UTC m=+1014.968127056"
Sep 17 03:36:05.139102 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:05.139072    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:05.144957 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:05.144932    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:05.347362 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:05.347338    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:05.348156 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:05.348137    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:14.143670 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:14.143640    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"]
Sep 17 03:36:14.149461 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:14.149430    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"
Sep 17 03:36:14.152648 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:14.152625    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cdaa7ee2-dockercfg-fd64v\""
Sep 17 03:36:14.152782 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:14.152766    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cdaa7ee2\""
Sep 17 03:36:14.158723 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:14.158701    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"]
Sep 17 03:36:14.216648 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:14.216607    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b82160cb-f341-4b61-b41a-1a67bb0ed248-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg\" (UID: \"b82160cb-f341-4b61-b41a-1a67bb0ed248\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"
Sep 17 03:36:14.216786 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:14.216750    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkdqg\" (UniqueName: \"kubernetes.io/projected/b82160cb-f341-4b61-b41a-1a67bb0ed248-kube-api-access-vkdqg\") pod \"payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg\" (UID: \"b82160cb-f341-4b61-b41a-1a67bb0ed248\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"
Sep 17 03:36:14.317451 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:14.317412    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b82160cb-f341-4b61-b41a-1a67bb0ed248-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg\" (UID: \"b82160cb-f341-4b61-b41a-1a67bb0ed248\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"
Sep 17 03:36:14.317625 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:14.317507    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkdqg\" (UniqueName: \"kubernetes.io/projected/b82160cb-f341-4b61-b41a-1a67bb0ed248-kube-api-access-vkdqg\") pod \"payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg\" (UID: \"b82160cb-f341-4b61-b41a-1a67bb0ed248\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"
Sep 17 03:36:14.318103 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:14.318083    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b82160cb-f341-4b61-b41a-1a67bb0ed248-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg\" (UID: \"b82160cb-f341-4b61-b41a-1a67bb0ed248\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"
Sep 17 03:36:14.327654 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:14.327626    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkdqg\" (UniqueName: \"kubernetes.io/projected/b82160cb-f341-4b61-b41a-1a67bb0ed248-kube-api-access-vkdqg\") pod \"payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg\" (UID: \"b82160cb-f341-4b61-b41a-1a67bb0ed248\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"
Sep 17 03:36:14.461715 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:14.461643    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"
Sep 17 03:36:14.737220 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:14.737178    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"]
Sep 17 03:36:14.738761 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:36:14.738737    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82160cb_f341_4b61_b41a_1a67bb0ed248.slice/crio-5a607e449ac382b61fa3ebb2e5257200118b61b944ef5dcdc9e67243915631bd WatchSource:0}: Error finding container 5a607e449ac382b61fa3ebb2e5257200118b61b944ef5dcdc9e67243915631bd: Status 404 returned error can't find the container with id 5a607e449ac382b61fa3ebb2e5257200118b61b944ef5dcdc9e67243915631bd
Sep 17 03:36:15.405696 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:15.405650    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg" event={"ID":"b82160cb-f341-4b61-b41a-1a67bb0ed248","Type":"ContainerStarted","Data":"e1d6490045e64fee5f82f7abdb9dccef454ee308cb3247e7b2b039e4fa368470"}
Sep 17 03:36:15.406142 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:15.405702    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg" event={"ID":"b82160cb-f341-4b61-b41a-1a67bb0ed248","Type":"ContainerStarted","Data":"5a607e449ac382b61fa3ebb2e5257200118b61b944ef5dcdc9e67243915631bd"}
Sep 17 03:36:15.406142 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:15.405730    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"
Sep 17 03:36:15.427890 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:15.427834    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg" podStartSLOduration=1.4278135619999999 podStartE2EDuration="1.427813562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:36:15.426069726 +0000 UTC m=+1026.029165731" watchObservedRunningTime="2026-09-17 03:36:15.427813562 +0000 UTC m=+1026.030909571"
Sep 17 03:36:19.376953 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.376922    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"]
Sep 17 03:36:19.377468 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.377173    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj" podUID="2b557adf-444c-4a8d-8c2e-0fe3a6faf348" containerName="maas-api" containerID="cri-o://2dd39243a7dd7ce4220c38a5406d4b5f7d3628eb4cee9140acb58a6c6bfe0ae1" gracePeriod=30
Sep 17 03:36:19.440009 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.439976    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"]
Sep 17 03:36:19.440245 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.440214    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6" podUID="22c1d400-a6ba-41f2-ab48-9d45c5887ab8" containerName="payload-pre-processing" containerID="cri-o://fdc14dc54f671380e19569e30c3d043dbbde7af6de6ee767ec90afa0e5a3d840" gracePeriod=30
Sep 17 03:36:19.756740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.756676    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:36:19.774964 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.774928    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"
Sep 17 03:36:19.873971 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.873943    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c1d400-a6ba-41f2-ab48-9d45c5887ab8-plugins-config-volume\") pod \"22c1d400-a6ba-41f2-ab48-9d45c5887ab8\" (UID: \"22c1d400-a6ba-41f2-ab48-9d45c5887ab8\") "
Sep 17 03:36:19.874136 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.873976    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-metrics-certs\") pod \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") "
Sep 17 03:36:19.874136 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.874107    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52snl\" (UniqueName: \"kubernetes.io/projected/22c1d400-a6ba-41f2-ab48-9d45c5887ab8-kube-api-access-52snl\") pod \"22c1d400-a6ba-41f2-ab48-9d45c5887ab8\" (UID: \"22c1d400-a6ba-41f2-ab48-9d45c5887ab8\") "
Sep 17 03:36:19.874136 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.874127    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h226g\" (UniqueName: \"kubernetes.io/projected/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-kube-api-access-h226g\") pod \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") "
Sep 17 03:36:19.874322 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.874148    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-openshift-service-ca\") pod \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") "
Sep 17 03:36:19.874322 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.874163    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-maas-api-tls\") pod \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\" (UID: \"2b557adf-444c-4a8d-8c2e-0fe3a6faf348\") "
Sep 17 03:36:19.874490 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.874322    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c1d400-a6ba-41f2-ab48-9d45c5887ab8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22c1d400-a6ba-41f2-ab48-9d45c5887ab8" (UID: "22c1d400-a6ba-41f2-ab48-9d45c5887ab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:36:19.874674 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.874649    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2b557adf-444c-4a8d-8c2e-0fe3a6faf348" (UID: "2b557adf-444c-4a8d-8c2e-0fe3a6faf348"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:36:19.874750 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.874677    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c1d400-a6ba-41f2-ab48-9d45c5887ab8-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:19.876226 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.876185    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2b557adf-444c-4a8d-8c2e-0fe3a6faf348" (UID: "2b557adf-444c-4a8d-8c2e-0fe3a6faf348"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:36:19.876325 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.876230    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c1d400-a6ba-41f2-ab48-9d45c5887ab8-kube-api-access-52snl" (OuterVolumeSpecName: "kube-api-access-52snl") pod "22c1d400-a6ba-41f2-ab48-9d45c5887ab8" (UID: "22c1d400-a6ba-41f2-ab48-9d45c5887ab8"). InnerVolumeSpecName "kube-api-access-52snl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:36:19.876325 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.876246    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-kube-api-access-h226g" (OuterVolumeSpecName: "kube-api-access-h226g") pod "2b557adf-444c-4a8d-8c2e-0fe3a6faf348" (UID: "2b557adf-444c-4a8d-8c2e-0fe3a6faf348"). InnerVolumeSpecName "kube-api-access-h226g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:36:19.876479 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.876465    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2b557adf-444c-4a8d-8c2e-0fe3a6faf348" (UID: "2b557adf-444c-4a8d-8c2e-0fe3a6faf348"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:36:19.975226 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.975164    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52snl\" (UniqueName: \"kubernetes.io/projected/22c1d400-a6ba-41f2-ab48-9d45c5887ab8-kube-api-access-52snl\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:19.975226 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.975203    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h226g\" (UniqueName: \"kubernetes.io/projected/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-kube-api-access-h226g\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:19.975226 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.975218    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-openshift-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:19.975396 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.975230    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-maas-api-tls\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:19.975396 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:19.975242    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b557adf-444c-4a8d-8c2e-0fe3a6faf348-metrics-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:20.438499 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.438470    2557 generic.go:356] "Generic (PLEG): container finished" podID="2b557adf-444c-4a8d-8c2e-0fe3a6faf348" containerID="2dd39243a7dd7ce4220c38a5406d4b5f7d3628eb4cee9140acb58a6c6bfe0ae1" exitCode=0
Sep 17 03:36:20.438909 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.438514    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"
Sep 17 03:36:20.438909 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.438551    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj" event={"ID":"2b557adf-444c-4a8d-8c2e-0fe3a6faf348","Type":"ContainerDied","Data":"2dd39243a7dd7ce4220c38a5406d4b5f7d3628eb4cee9140acb58a6c6bfe0ae1"}
Sep 17 03:36:20.438909 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.438598    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj" event={"ID":"2b557adf-444c-4a8d-8c2e-0fe3a6faf348","Type":"ContainerDied","Data":"71db437460157f7997ecab40408989128cb12a9f8b180b7d9a79c8dbc405bccb"}
Sep 17 03:36:20.438909 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.438621    2557 scope.go:117] "RemoveContainer" containerID="2dd39243a7dd7ce4220c38a5406d4b5f7d3628eb4cee9140acb58a6c6bfe0ae1"
Sep 17 03:36:20.440082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.440056    2557 generic.go:356] "Generic (PLEG): container finished" podID="22c1d400-a6ba-41f2-ab48-9d45c5887ab8" containerID="fdc14dc54f671380e19569e30c3d043dbbde7af6de6ee767ec90afa0e5a3d840" exitCode=0
Sep 17 03:36:20.440166 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.440090    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6" event={"ID":"22c1d400-a6ba-41f2-ab48-9d45c5887ab8","Type":"ContainerDied","Data":"fdc14dc54f671380e19569e30c3d043dbbde7af6de6ee767ec90afa0e5a3d840"}
Sep 17 03:36:20.440166 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.440109    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"
Sep 17 03:36:20.440166 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.440135    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6" event={"ID":"22c1d400-a6ba-41f2-ab48-9d45c5887ab8","Type":"ContainerDied","Data":"4d13fc020d7a88345da30214d23c43371973c5959b75c4952a9a058360fcccac"}
Sep 17 03:36:20.448263 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.448248    2557 scope.go:117] "RemoveContainer" containerID="2dd39243a7dd7ce4220c38a5406d4b5f7d3628eb4cee9140acb58a6c6bfe0ae1"
Sep 17 03:36:20.448548 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:36:20.448520    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd39243a7dd7ce4220c38a5406d4b5f7d3628eb4cee9140acb58a6c6bfe0ae1\": container with ID starting with 2dd39243a7dd7ce4220c38a5406d4b5f7d3628eb4cee9140acb58a6c6bfe0ae1 not found: ID does not exist" containerID="2dd39243a7dd7ce4220c38a5406d4b5f7d3628eb4cee9140acb58a6c6bfe0ae1"
Sep 17 03:36:20.448661 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.448554    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd39243a7dd7ce4220c38a5406d4b5f7d3628eb4cee9140acb58a6c6bfe0ae1"} err="failed to get container status \"2dd39243a7dd7ce4220c38a5406d4b5f7d3628eb4cee9140acb58a6c6bfe0ae1\": rpc error: code = NotFound desc = could not find container \"2dd39243a7dd7ce4220c38a5406d4b5f7d3628eb4cee9140acb58a6c6bfe0ae1\": container with ID starting with 2dd39243a7dd7ce4220c38a5406d4b5f7d3628eb4cee9140acb58a6c6bfe0ae1 not found: ID does not exist"
Sep 17 03:36:20.448661 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.448576    2557 scope.go:117] "RemoveContainer" containerID="fdc14dc54f671380e19569e30c3d043dbbde7af6de6ee767ec90afa0e5a3d840"
Sep 17 03:36:20.457359 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.457340    2557 scope.go:117] "RemoveContainer" containerID="fdc14dc54f671380e19569e30c3d043dbbde7af6de6ee767ec90afa0e5a3d840"
Sep 17 03:36:20.457652 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:36:20.457633    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc14dc54f671380e19569e30c3d043dbbde7af6de6ee767ec90afa0e5a3d840\": container with ID starting with fdc14dc54f671380e19569e30c3d043dbbde7af6de6ee767ec90afa0e5a3d840 not found: ID does not exist" containerID="fdc14dc54f671380e19569e30c3d043dbbde7af6de6ee767ec90afa0e5a3d840"
Sep 17 03:36:20.457720 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.457660    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc14dc54f671380e19569e30c3d043dbbde7af6de6ee767ec90afa0e5a3d840"} err="failed to get container status \"fdc14dc54f671380e19569e30c3d043dbbde7af6de6ee767ec90afa0e5a3d840\": rpc error: code = NotFound desc = could not find container \"fdc14dc54f671380e19569e30c3d043dbbde7af6de6ee767ec90afa0e5a3d840\": container with ID starting with fdc14dc54f671380e19569e30c3d043dbbde7af6de6ee767ec90afa0e5a3d840 not found: ID does not exist"
Sep 17 03:36:20.457930 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.457913    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"]
Sep 17 03:36:20.462309 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.462290    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-5062bf-65b9d88b6-6t9b6"]
Sep 17 03:36:20.477228 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.477203    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"]
Sep 17 03:36:20.484610 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:20.484591    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-5062bf-78c665b5b-5pnvj"]
Sep 17 03:36:21.921727 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:21.921684    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22c1d400-a6ba-41f2-ab48-9d45c5887ab8" path="/var/lib/kubelet/pods/22c1d400-a6ba-41f2-ab48-9d45c5887ab8/volumes"
Sep 17 03:36:21.922320 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:21.922293    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b557adf-444c-4a8d-8c2e-0fe3a6faf348" path="/var/lib/kubelet/pods/2b557adf-444c-4a8d-8c2e-0fe3a6faf348/volumes"
Sep 17 03:36:26.367703 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.367656    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"]
Sep 17 03:36:26.368300 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.368278    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c1d400-a6ba-41f2-ab48-9d45c5887ab8" containerName="payload-pre-processing"
Sep 17 03:36:26.368445 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.368302    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c1d400-a6ba-41f2-ab48-9d45c5887ab8" containerName="payload-pre-processing"
Sep 17 03:36:26.368445 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.368316    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b557adf-444c-4a8d-8c2e-0fe3a6faf348" containerName="maas-api"
Sep 17 03:36:26.368445 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.368322    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b557adf-444c-4a8d-8c2e-0fe3a6faf348" containerName="maas-api"
Sep 17 03:36:26.368445 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.368433    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c1d400-a6ba-41f2-ab48-9d45c5887ab8" containerName="payload-pre-processing"
Sep 17 03:36:26.368445 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.368449    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b557adf-444c-4a8d-8c2e-0fe3a6faf348" containerName="maas-api"
Sep 17 03:36:26.373588 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.373561    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.376970 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.376938    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-214fea\"/\"openshift-service-ca.crt\""
Sep 17 03:36:26.377839 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.377816    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-214fea\"/\"default-dockercfg-g7kkn\""
Sep 17 03:36:26.378689 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.378664    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-214fea\"/\"kube-root-ca.crt\""
Sep 17 03:36:26.379131 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.379108    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-214fea\"/\"e2e-conflict-a-fed91830-kserve-self-signed-certs\""
Sep 17 03:36:26.382324 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.382305    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"]
Sep 17 03:36:26.413341 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.413323    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"
Sep 17 03:36:26.538158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.538124    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-model-cache\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.538330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.538187    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-tmp-dir\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.538330 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.538297    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca02225-cef7-4f9c-8545-6c07bf293a3e-tls-certs\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.538466 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.538441    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwrl4\" (UniqueName: \"kubernetes.io/projected/aca02225-cef7-4f9c-8545-6c07bf293a3e-kube-api-access-jwrl4\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.538511 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.538496    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-home\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.538550 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.538536    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-dshm\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.639866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.639794    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-model-cache\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.639866 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.639830    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-tmp-dir\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.640075 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.639872    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca02225-cef7-4f9c-8545-6c07bf293a3e-tls-certs\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.640075 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.639891    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwrl4\" (UniqueName: \"kubernetes.io/projected/aca02225-cef7-4f9c-8545-6c07bf293a3e-kube-api-access-jwrl4\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.640075 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.639912    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-home\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.640075 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.639926    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-dshm\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.640333 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.640246    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-model-cache\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.640333 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.640312    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-tmp-dir\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.640415 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.640334    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-home\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.641997 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.641978    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-dshm\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.642317 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.642297    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca02225-cef7-4f9c-8545-6c07bf293a3e-tls-certs\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.647411 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.647389    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwrl4\" (UniqueName: \"kubernetes.io/projected/aca02225-cef7-4f9c-8545-6c07bf293a3e-kube-api-access-jwrl4\") pod \"e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") " pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:26.688430 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:26.688404    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:27.004508 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:27.004484    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"]
Sep 17 03:36:27.006901 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:36:27.006868    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaca02225_cef7_4f9c_8545_6c07bf293a3e.slice/crio-5c350b3e16fc080787856a3d202e31b5e75252294e6bd1df9a651bf77357314b WatchSource:0}: Error finding container 5c350b3e16fc080787856a3d202e31b5e75252294e6bd1df9a651bf77357314b: Status 404 returned error can't find the container with id 5c350b3e16fc080787856a3d202e31b5e75252294e6bd1df9a651bf77357314b
Sep 17 03:36:27.484726 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:27.484688    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm" event={"ID":"aca02225-cef7-4f9c-8545-6c07bf293a3e","Type":"ContainerStarted","Data":"4d2f981bd5dc73f066efa6d1e18c9fe94078161324783065a6f61a1b5339a3e9"}
Sep 17 03:36:27.485128 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:27.484739    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm" event={"ID":"aca02225-cef7-4f9c-8545-6c07bf293a3e","Type":"ContainerStarted","Data":"5c350b3e16fc080787856a3d202e31b5e75252294e6bd1df9a651bf77357314b"}
Sep 17 03:36:27.505445 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:27.505401    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm" podStartSLOduration=1.505385727 podStartE2EDuration="1.505385727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:36:27.50426389 +0000 UTC m=+1038.107359899" watchObservedRunningTime="2026-09-17 03:36:27.505385727 +0000 UTC m=+1038.108481793"
Sep 17 03:36:36.389405 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:36.389353    2557 patch_prober.go:28] interesting pod/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 03:36:36.389929 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:36.389455    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj" podUID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 03:36:36.689176 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:36.689098    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:36.689176 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:36.689150    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:36.702277 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:36.702255    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:37.563161 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:37.563133    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:36:38.559078 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:38.559007    2557 generic.go:356] "Generic (PLEG): container finished" podID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" containerID="040ba2507e949b1388569f935f5c3e7efac6cec007b841f565e58250bb686369" exitCode=0
Sep 17 03:36:38.559078 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:38.559051    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj" event={"ID":"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff","Type":"ContainerDied","Data":"040ba2507e949b1388569f935f5c3e7efac6cec007b841f565e58250bb686369"}
Sep 17 03:36:38.559539 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:38.559522    2557 scope.go:117] "RemoveContainer" containerID="040ba2507e949b1388569f935f5c3e7efac6cec007b841f565e58250bb686369"
Sep 17 03:36:38.560819 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:38.560791    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:36:38.560889 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:38.560859    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:36:38.560930 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:38.560892    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:36:39.566413 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:39.566378    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj" event={"ID":"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff","Type":"ContainerStarted","Data":"1316c6acee6a0a8826a54953725b052dd56f45924ca6b77ac13a1673753e8516"}
Sep 17 03:36:40.138969 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:40.138930    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:40.139253 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:40.139224    2557 patch_prober.go:28] interesting pod/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 17 03:36:40.139354 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:40.139289    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj" podUID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 17 03:36:41.050215 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.037048    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"]
Sep 17 03:36:41.050215 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.037316    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg" podUID="b82160cb-f341-4b61-b41a-1a67bb0ed248" containerName="payload-pre-processing" containerID="cri-o://e1d6490045e64fee5f82f7abdb9dccef454ee308cb3247e7b2b039e4fa368470" gracePeriod=30
Sep 17 03:36:41.146891 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.139675    2557 patch_prober.go:28] interesting pod/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 17 03:36:41.146891 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.139758    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj" podUID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 17 03:36:41.355219 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.354477    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"
Sep 17 03:36:41.394252 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.393557    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b82160cb-f341-4b61-b41a-1a67bb0ed248-plugins-config-volume\") pod \"b82160cb-f341-4b61-b41a-1a67bb0ed248\" (UID: \"b82160cb-f341-4b61-b41a-1a67bb0ed248\") "
Sep 17 03:36:41.394252 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.393757    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkdqg\" (UniqueName: \"kubernetes.io/projected/b82160cb-f341-4b61-b41a-1a67bb0ed248-kube-api-access-vkdqg\") pod \"b82160cb-f341-4b61-b41a-1a67bb0ed248\" (UID: \"b82160cb-f341-4b61-b41a-1a67bb0ed248\") "
Sep 17 03:36:41.394499 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.394374    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82160cb-f341-4b61-b41a-1a67bb0ed248-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b82160cb-f341-4b61-b41a-1a67bb0ed248" (UID: "b82160cb-f341-4b61-b41a-1a67bb0ed248"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:36:41.396105 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.396071    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82160cb-f341-4b61-b41a-1a67bb0ed248-kube-api-access-vkdqg" (OuterVolumeSpecName: "kube-api-access-vkdqg") pod "b82160cb-f341-4b61-b41a-1a67bb0ed248" (UID: "b82160cb-f341-4b61-b41a-1a67bb0ed248"). InnerVolumeSpecName "kube-api-access-vkdqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:36:41.495419 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.495388    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkdqg\" (UniqueName: \"kubernetes.io/projected/b82160cb-f341-4b61-b41a-1a67bb0ed248-kube-api-access-vkdqg\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:41.495419 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.495413    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b82160cb-f341-4b61-b41a-1a67bb0ed248-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:41.579387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.579302    2557 generic.go:356] "Generic (PLEG): container finished" podID="b82160cb-f341-4b61-b41a-1a67bb0ed248" containerID="e1d6490045e64fee5f82f7abdb9dccef454ee308cb3247e7b2b039e4fa368470" exitCode=0
Sep 17 03:36:41.579387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.579349    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"
Sep 17 03:36:41.579609 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.579390    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg" event={"ID":"b82160cb-f341-4b61-b41a-1a67bb0ed248","Type":"ContainerDied","Data":"e1d6490045e64fee5f82f7abdb9dccef454ee308cb3247e7b2b039e4fa368470"}
Sep 17 03:36:41.579609 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.579448    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg" event={"ID":"b82160cb-f341-4b61-b41a-1a67bb0ed248","Type":"ContainerDied","Data":"5a607e449ac382b61fa3ebb2e5257200118b61b944ef5dcdc9e67243915631bd"}
Sep 17 03:36:41.579609 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.579482    2557 scope.go:117] "RemoveContainer" containerID="e1d6490045e64fee5f82f7abdb9dccef454ee308cb3247e7b2b039e4fa368470"
Sep 17 03:36:41.589568 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.589550    2557 scope.go:117] "RemoveContainer" containerID="e1d6490045e64fee5f82f7abdb9dccef454ee308cb3247e7b2b039e4fa368470"
Sep 17 03:36:41.589851 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:36:41.589830    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d6490045e64fee5f82f7abdb9dccef454ee308cb3247e7b2b039e4fa368470\": container with ID starting with e1d6490045e64fee5f82f7abdb9dccef454ee308cb3247e7b2b039e4fa368470 not found: ID does not exist" containerID="e1d6490045e64fee5f82f7abdb9dccef454ee308cb3247e7b2b039e4fa368470"
Sep 17 03:36:41.589914 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.589859    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d6490045e64fee5f82f7abdb9dccef454ee308cb3247e7b2b039e4fa368470"} err="failed to get container status \"e1d6490045e64fee5f82f7abdb9dccef454ee308cb3247e7b2b039e4fa368470\": rpc error: code = NotFound desc = could not find container \"e1d6490045e64fee5f82f7abdb9dccef454ee308cb3247e7b2b039e4fa368470\": container with ID starting with e1d6490045e64fee5f82f7abdb9dccef454ee308cb3247e7b2b039e4fa368470 not found: ID does not exist"
Sep 17 03:36:41.601971 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.601944    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"]
Sep 17 03:36:41.607281 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.607258    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cdaa7ee2-8695f9778-qnlzg"]
Sep 17 03:36:41.916315 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:41.916284    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b82160cb-f341-4b61-b41a-1a67bb0ed248" path="/var/lib/kubelet/pods/b82160cb-f341-4b61-b41a-1a67bb0ed248/volumes"
Sep 17 03:36:42.139603 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:42.139571    2557 patch_prober.go:28] interesting pod/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 17 03:36:42.140206 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:42.140151    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj" podUID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 17 03:36:43.140266 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:43.140214    2557 patch_prober.go:28] interesting pod/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 17 03:36:43.140769 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:43.140289    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj" podUID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 17 03:36:44.139504 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:44.139471    2557 patch_prober.go:28] interesting pod/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Sep 17 03:36:44.139763 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:44.139733    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj" podUID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Sep 17 03:36:45.142608 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:45.142574    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:45.143049 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:45.142971    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:45.143730 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:45.143710    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:47.985171 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:47.985139    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"]
Sep 17 03:36:47.985690 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:47.985389    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj" podUID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" containerName="istio-proxy" containerID="cri-o://1316c6acee6a0a8826a54953725b052dd56f45924ca6b77ac13a1673753e8516" gracePeriod=30
Sep 17 03:36:53.237035 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.237012    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:53.309814 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.309755    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-token\") pod \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") "
Sep 17 03:36:53.309814 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.309813    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istiod-ca-cert\") pod \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") "
Sep 17 03:36:53.309993 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.309843    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-workload-socket\") pod \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") "
Sep 17 03:36:53.309993 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.309867    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-workload-certs\") pod \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") "
Sep 17 03:36:53.309993 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.309888    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-data\") pod \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") "
Sep 17 03:36:53.309993 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.309904    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-podinfo\") pod \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") "
Sep 17 03:36:53.309993 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.309931    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-credential-socket\") pod \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") "
Sep 17 03:36:53.309993 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.309954    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-envoy\") pod \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") "
Sep 17 03:36:53.310339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.310029    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhcg5\" (UniqueName: \"kubernetes.io/projected/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-kube-api-access-xhcg5\") pod \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\" (UID: \"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff\") "
Sep 17 03:36:53.310339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.310221    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" (UID: "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:36:53.310339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.310273    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" (UID: "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:36:53.310339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.310327    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" (UID: "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:36:53.310559 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.310511    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-data" (OuterVolumeSpecName: "istio-data") pod "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" (UID: "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:36:53.310634 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.310572    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" (UID: "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:36:53.310634 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.310582    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:53.310634 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.310620    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:53.310792 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.310636    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:53.311972 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.311947    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" (UID: "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:36:53.312072 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.312050    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" (UID: "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:36:53.312288 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.312271    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" (UID: "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:36:53.312344 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.312284    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-kube-api-access-xhcg5" (OuterVolumeSpecName: "kube-api-access-xhcg5") pod "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" (UID: "b9ff71cb-9f6e-494f-b08b-62ecce70c0ff"). InnerVolumeSpecName "kube-api-access-xhcg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:36:53.411917 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.411895    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:53.411917 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.411917    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:53.412043 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.411927    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:53.412043 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.411936    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:53.412043 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.411945    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:53.412043 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.411953    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhcg5\" (UniqueName: \"kubernetes.io/projected/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff-kube-api-access-xhcg5\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:36:53.649463 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.649435    2557 generic.go:356] "Generic (PLEG): container finished" podID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" containerID="1316c6acee6a0a8826a54953725b052dd56f45924ca6b77ac13a1673753e8516" exitCode=0
Sep 17 03:36:53.649557 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.649487    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"
Sep 17 03:36:53.649557 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.649524    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj" event={"ID":"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff","Type":"ContainerDied","Data":"1316c6acee6a0a8826a54953725b052dd56f45924ca6b77ac13a1673753e8516"}
Sep 17 03:36:53.649635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.649579    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj" event={"ID":"b9ff71cb-9f6e-494f-b08b-62ecce70c0ff","Type":"ContainerDied","Data":"c90252722e9422dc038b3980ed656be3eb43d2eefa3f21a44d922f67ee962f97"}
Sep 17 03:36:53.649635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.649604    2557 scope.go:117] "RemoveContainer" containerID="1316c6acee6a0a8826a54953725b052dd56f45924ca6b77ac13a1673753e8516"
Sep 17 03:36:53.659159 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.659142    2557 scope.go:117] "RemoveContainer" containerID="040ba2507e949b1388569f935f5c3e7efac6cec007b841f565e58250bb686369"
Sep 17 03:36:53.667313 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.667295    2557 scope.go:117] "RemoveContainer" containerID="1316c6acee6a0a8826a54953725b052dd56f45924ca6b77ac13a1673753e8516"
Sep 17 03:36:53.667668 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:36:53.667632    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1316c6acee6a0a8826a54953725b052dd56f45924ca6b77ac13a1673753e8516\": container with ID starting with 1316c6acee6a0a8826a54953725b052dd56f45924ca6b77ac13a1673753e8516 not found: ID does not exist" containerID="1316c6acee6a0a8826a54953725b052dd56f45924ca6b77ac13a1673753e8516"
Sep 17 03:36:53.667770 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.667669    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1316c6acee6a0a8826a54953725b052dd56f45924ca6b77ac13a1673753e8516"} err="failed to get container status \"1316c6acee6a0a8826a54953725b052dd56f45924ca6b77ac13a1673753e8516\": rpc error: code = NotFound desc = could not find container \"1316c6acee6a0a8826a54953725b052dd56f45924ca6b77ac13a1673753e8516\": container with ID starting with 1316c6acee6a0a8826a54953725b052dd56f45924ca6b77ac13a1673753e8516 not found: ID does not exist"
Sep 17 03:36:53.667770 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.667692    2557 scope.go:117] "RemoveContainer" containerID="040ba2507e949b1388569f935f5c3e7efac6cec007b841f565e58250bb686369"
Sep 17 03:36:53.668047 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:36:53.667994    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"040ba2507e949b1388569f935f5c3e7efac6cec007b841f565e58250bb686369\": container with ID starting with 040ba2507e949b1388569f935f5c3e7efac6cec007b841f565e58250bb686369 not found: ID does not exist" containerID="040ba2507e949b1388569f935f5c3e7efac6cec007b841f565e58250bb686369"
Sep 17 03:36:53.668047 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.668037    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040ba2507e949b1388569f935f5c3e7efac6cec007b841f565e58250bb686369"} err="failed to get container status \"040ba2507e949b1388569f935f5c3e7efac6cec007b841f565e58250bb686369\": rpc error: code = NotFound desc = could not find container \"040ba2507e949b1388569f935f5c3e7efac6cec007b841f565e58250bb686369\": container with ID starting with 040ba2507e949b1388569f935f5c3e7efac6cec007b841f565e58250bb686369 not found: ID does not exist"
Sep 17 03:36:53.670470 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.670449    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"]
Sep 17 03:36:53.673616 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.673596    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cdaa7ee2-openshift-default-86c79d4c64-tk4hj"]
Sep 17 03:36:53.915644 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:36:53.915594    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" path="/var/lib/kubelet/pods/b9ff71cb-9f6e-494f-b08b-62ecce70c0ff/volumes"
Sep 17 03:37:00.780287 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.780257    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"]
Sep 17 03:37:00.780813 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.780794    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" containerName="istio-proxy"
Sep 17 03:37:00.780813 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.780815    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" containerName="istio-proxy"
Sep 17 03:37:00.780897 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.780827    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" containerName="istio-proxy"
Sep 17 03:37:00.780897 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.780832    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" containerName="istio-proxy"
Sep 17 03:37:00.780897 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.780844    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b82160cb-f341-4b61-b41a-1a67bb0ed248" containerName="payload-pre-processing"
Sep 17 03:37:00.780897 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.780850    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b82160cb-f341-4b61-b41a-1a67bb0ed248" containerName="payload-pre-processing"
Sep 17 03:37:00.781021 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.780918    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" containerName="istio-proxy"
Sep 17 03:37:00.781021 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.780926    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b82160cb-f341-4b61-b41a-1a67bb0ed248" containerName="payload-pre-processing"
Sep 17 03:37:00.781021 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.780933    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9ff71cb-9f6e-494f-b08b-62ecce70c0ff" containerName="istio-proxy"
Sep 17 03:37:00.785374 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.785358    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.787423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.787398    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1afd34\"/\"openshift-service-ca.crt\""
Sep 17 03:37:00.787505 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.787471    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1afd34\"/\"kube-root-ca.crt\""
Sep 17 03:37:00.788038 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.788021    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1afd34\"/\"default-dockercfg-29l8s\""
Sep 17 03:37:00.788124 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.788058    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1afd34\"/\"distinct-w0-1afd34-kserve-self-signed-certs\""
Sep 17 03:37:00.793938 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.793916    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"]
Sep 17 03:37:00.881129 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.881099    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-dshm\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.881283 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.881168    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-tmp-dir\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.881283 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.881210    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97fbb\" (UniqueName: \"kubernetes.io/projected/c4a8a698-79e4-48ad-984b-a1709305ff8c-kube-api-access-97fbb\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.881283 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.881263    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-home\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.881469 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.881298    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-model-cache\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.881469 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.881323    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a8a698-79e4-48ad-984b-a1709305ff8c-tls-certs\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.982952 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.982049    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-home\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.982952 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.982097    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-model-cache\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.982952 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.982155    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a8a698-79e4-48ad-984b-a1709305ff8c-tls-certs\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.982952 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.982238    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-dshm\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.982952 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.982335    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-tmp-dir\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.982952 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.982360    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97fbb\" (UniqueName: \"kubernetes.io/projected/c4a8a698-79e4-48ad-984b-a1709305ff8c-kube-api-access-97fbb\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.982952 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.982523    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-home\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.983516 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.983035    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-model-cache\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.983516 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.983304    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-tmp-dir\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.986275 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.986251    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a8a698-79e4-48ad-984b-a1709305ff8c-tls-certs\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.994024 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.993679    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-dshm\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:00.994792 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:00.994744    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97fbb\" (UniqueName: \"kubernetes.io/projected/c4a8a698-79e4-48ad-984b-a1709305ff8c-kube-api-access-97fbb\") pod \"distinct-w0-1afd34-kserve-6d7b57d648-t29zt\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") " pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:01.094891 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.094831    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:01.369092 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.369067    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"]
Sep 17 03:37:01.370749 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:37:01.370722    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4a8a698_79e4_48ad_984b_a1709305ff8c.slice/crio-344fbb63340e848a53ec9d3213d8d0e98e009616759f31c73ecb53b3bcf25088 WatchSource:0}: Error finding container 344fbb63340e848a53ec9d3213d8d0e98e009616759f31c73ecb53b3bcf25088: Status 404 returned error can't find the container with id 344fbb63340e848a53ec9d3213d8d0e98e009616759f31c73ecb53b3bcf25088
Sep 17 03:37:01.420766 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.418768    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"]
Sep 17 03:37:01.423829 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.423808    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.426341 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.426313    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-57958f-openshift-default-dockercfg-mc2z6\""
Sep 17 03:37:01.434443 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.434421    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"]
Sep 17 03:37:01.589815 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.589776    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-podinfo\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.590016 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.589836    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-workload-socket\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.590016 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.589926    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-data\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.590016 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.589963    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-token\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.590016 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.589993    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-envoy\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.590272 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.590060    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-workload-certs\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.590272 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.590096    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9d4t\" (UniqueName: \"kubernetes.io/projected/4a823a95-f61d-40c9-aaca-238000c3e77e-kube-api-access-r9d4t\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.590272 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.590227    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a823a95-f61d-40c9-aaca-238000c3e77e-istiod-ca-cert\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.590436 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.590271    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-credential-socket\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.691188 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.691123    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a823a95-f61d-40c9-aaca-238000c3e77e-istiod-ca-cert\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.691188 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.691161    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-credential-socket\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.691391 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.691205    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-podinfo\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.691391 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.691250    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-workload-socket\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.691391 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.691286    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-data\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.691391 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.691311    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-token\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.691391 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.691348    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-envoy\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.691391 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.691384    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-workload-certs\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.691717 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.691417    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9d4t\" (UniqueName: \"kubernetes.io/projected/4a823a95-f61d-40c9-aaca-238000c3e77e-kube-api-access-r9d4t\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.691717 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.691585    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-credential-socket\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.691840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.691752    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-workload-socket\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.691840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.691779    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-data\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.691982 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.691960    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a823a95-f61d-40c9-aaca-238000c3e77e-istiod-ca-cert\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.692174 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.692143    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-workload-certs\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.693873 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.693854    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-envoy\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.694240 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.694185    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-podinfo\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.698357 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.698324    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt" event={"ID":"c4a8a698-79e4-48ad-984b-a1709305ff8c","Type":"ContainerStarted","Data":"16fb21e53a020a484644993c2860ac91511b82f1ce804d9c1b7d51b9c8b7341e"}
Sep 17 03:37:01.698460 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.698372    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt" event={"ID":"c4a8a698-79e4-48ad-984b-a1709305ff8c","Type":"ContainerStarted","Data":"344fbb63340e848a53ec9d3213d8d0e98e009616759f31c73ecb53b3bcf25088"}
Sep 17 03:37:01.698940 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.698922    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-token\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.699150 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.699133    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9d4t\" (UniqueName: \"kubernetes.io/projected/4a823a95-f61d-40c9-aaca-238000c3e77e-kube-api-access-r9d4t\") pod \"e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") " pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:01.718058 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.718022    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt" podStartSLOduration=1.718010122 podStartE2EDuration="1.718010122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:37:01.716883934 +0000 UTC m=+1072.319979938" watchObservedRunningTime="2026-09-17 03:37:01.718010122 +0000 UTC m=+1072.321106125"
Sep 17 03:37:01.737373 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:01.737352    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:02.011858 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:02.011832    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"]
Sep 17 03:37:02.013657 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:37:02.013620    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a823a95_f61d_40c9_aaca_238000c3e77e.slice/crio-d6ee543b7ba3b2d584e1a09c7d0eee66d565f47865cf5f12462e1de44b108322 WatchSource:0}: Error finding container d6ee543b7ba3b2d584e1a09c7d0eee66d565f47865cf5f12462e1de44b108322: Status 404 returned error can't find the container with id d6ee543b7ba3b2d584e1a09c7d0eee66d565f47865cf5f12462e1de44b108322
Sep 17 03:37:02.015785 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:02.015753    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:37:02.015856 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:02.015822    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:37:02.015856 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:02.015849    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:37:02.622075 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:02.622039    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"]
Sep 17 03:37:02.622311 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:02.622265    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h" podUID="72ba5071-edfb-4133-836b-531abf633199" containerName="istio-proxy" containerID="cri-o://6214dfb9284e0dc7a9ebde4b39b42ca1a4ed1ddfacdd9a4cfc3db2e0bc156e72" gracePeriod=30
Sep 17 03:37:02.706237 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:02.706205    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm" event={"ID":"4a823a95-f61d-40c9-aaca-238000c3e77e","Type":"ContainerStarted","Data":"5e31008dc9ce71df6f50cff759cb26eea9ca0e1d35612e0bc0ce6225b300a4bf"}
Sep 17 03:37:02.706353 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:02.706261    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm" event={"ID":"4a823a95-f61d-40c9-aaca-238000c3e77e","Type":"ContainerStarted","Data":"d6ee543b7ba3b2d584e1a09c7d0eee66d565f47865cf5f12462e1de44b108322"}
Sep 17 03:37:02.727251 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:02.727171    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm" podStartSLOduration=1.727148241 podStartE2EDuration="1.727148241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:37:02.726123192 +0000 UTC m=+1073.329219209" watchObservedRunningTime="2026-09-17 03:37:02.727148241 +0000 UTC m=+1073.330244246"
Sep 17 03:37:02.737590 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:02.737551    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:03.744586 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:03.744556    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:04.717753 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:04.717725    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:04.718840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:04.718808    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:07.872027 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.872004    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:37:07.962129 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962061    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-workload-socket\") pod \"72ba5071-edfb-4133-836b-531abf633199\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") "
Sep 17 03:37:07.962129 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962105    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-workload-certs\") pod \"72ba5071-edfb-4133-836b-531abf633199\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") "
Sep 17 03:37:07.962335 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962134    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dcrt\" (UniqueName: \"kubernetes.io/projected/72ba5071-edfb-4133-836b-531abf633199-kube-api-access-6dcrt\") pod \"72ba5071-edfb-4133-836b-531abf633199\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") "
Sep 17 03:37:07.962335 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962162    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-istio-data\") pod \"72ba5071-edfb-4133-836b-531abf633199\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") "
Sep 17 03:37:07.962335 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962235    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-istio-envoy\") pod \"72ba5071-edfb-4133-836b-531abf633199\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") "
Sep 17 03:37:07.962335 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962261    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ba5071-edfb-4133-836b-531abf633199-istio-token\") pod \"72ba5071-edfb-4133-836b-531abf633199\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") "
Sep 17 03:37:07.962335 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962301    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ba5071-edfb-4133-836b-531abf633199-istiod-ca-cert\") pod \"72ba5071-edfb-4133-836b-531abf633199\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") "
Sep 17 03:37:07.962335 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962331    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ba5071-edfb-4133-836b-531abf633199-istio-podinfo\") pod \"72ba5071-edfb-4133-836b-531abf633199\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") "
Sep 17 03:37:07.962663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962356    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-credential-socket\") pod \"72ba5071-edfb-4133-836b-531abf633199\" (UID: \"72ba5071-edfb-4133-836b-531abf633199\") "
Sep 17 03:37:07.962663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962359    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "72ba5071-edfb-4133-836b-531abf633199" (UID: "72ba5071-edfb-4133-836b-531abf633199"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:37:07.962663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962409    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "72ba5071-edfb-4133-836b-531abf633199" (UID: "72ba5071-edfb-4133-836b-531abf633199"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:37:07.962827 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962669    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-istio-data" (OuterVolumeSpecName: "istio-data") pod "72ba5071-edfb-4133-836b-531abf633199" (UID: "72ba5071-edfb-4133-836b-531abf633199"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:37:07.962827 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962714    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ba5071-edfb-4133-836b-531abf633199-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "72ba5071-edfb-4133-836b-531abf633199" (UID: "72ba5071-edfb-4133-836b-531abf633199"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:37:07.962827 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962822    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:37:07.962988 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962840    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:37:07.962988 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962854    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72ba5071-edfb-4133-836b-531abf633199-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:37:07.962988 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.962867    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:37:07.963341 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.963315    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "72ba5071-edfb-4133-836b-531abf633199" (UID: "72ba5071-edfb-4133-836b-531abf633199"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:37:07.964722 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.964696    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "72ba5071-edfb-4133-836b-531abf633199" (UID: "72ba5071-edfb-4133-836b-531abf633199"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:37:07.964850 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.964810    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ba5071-edfb-4133-836b-531abf633199-istio-token" (OuterVolumeSpecName: "istio-token") pod "72ba5071-edfb-4133-836b-531abf633199" (UID: "72ba5071-edfb-4133-836b-531abf633199"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:37:07.964850 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.964827    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72ba5071-edfb-4133-836b-531abf633199-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "72ba5071-edfb-4133-836b-531abf633199" (UID: "72ba5071-edfb-4133-836b-531abf633199"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:37:07.964938 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:07.964921    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ba5071-edfb-4133-836b-531abf633199-kube-api-access-6dcrt" (OuterVolumeSpecName: "kube-api-access-6dcrt") pod "72ba5071-edfb-4133-836b-531abf633199" (UID: "72ba5071-edfb-4133-836b-531abf633199"). InnerVolumeSpecName "kube-api-access-6dcrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:37:08.063725 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.063694    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dcrt\" (UniqueName: \"kubernetes.io/projected/72ba5071-edfb-4133-836b-531abf633199-kube-api-access-6dcrt\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:37:08.063725 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.063724    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:37:08.063895 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.063734    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72ba5071-edfb-4133-836b-531abf633199-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:37:08.063895 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.063742    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72ba5071-edfb-4133-836b-531abf633199-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:37:08.063895 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.063750    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72ba5071-edfb-4133-836b-531abf633199-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:37:08.579606 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.579573    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"]
Sep 17 03:37:08.580007 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.579995    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ba5071-edfb-4133-836b-531abf633199" containerName="istio-proxy"
Sep 17 03:37:08.580054 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.580008    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ba5071-edfb-4133-836b-531abf633199" containerName="istio-proxy"
Sep 17 03:37:08.580054 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.580022    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ba5071-edfb-4133-836b-531abf633199" containerName="istio-proxy"
Sep 17 03:37:08.580054 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.580027    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ba5071-edfb-4133-836b-531abf633199" containerName="istio-proxy"
Sep 17 03:37:08.580148 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.580113    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ba5071-edfb-4133-836b-531abf633199" containerName="istio-proxy"
Sep 17 03:37:08.580148 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.580122    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ba5071-edfb-4133-836b-531abf633199" containerName="istio-proxy"
Sep 17 03:37:08.583544 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.583528    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"
Sep 17 03:37:08.585868 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.585847    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-57958f-dockercfg-8zmdz\""
Sep 17 03:37:08.586048 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.586036    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-57958f\""
Sep 17 03:37:08.596732 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.596703    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"]
Sep 17 03:37:08.617202 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.617169    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"]
Sep 17 03:37:08.621664 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.621645    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"
Sep 17 03:37:08.624936 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.624911    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"]
Sep 17 03:37:08.670300 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.670264    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b01fd2c-9199-4288-b509-029da1d8b9a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v\" (UID: \"9b01fd2c-9199-4288-b509-029da1d8b9a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"
Sep 17 03:37:08.670488 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.670324    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf5rs\" (UniqueName: \"kubernetes.io/projected/0d11737d-b322-4bef-9c56-5d1bcb52d5cd-kube-api-access-qf5rs\") pod \"payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls\" (UID: \"0d11737d-b322-4bef-9c56-5d1bcb52d5cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"
Sep 17 03:37:08.670488 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.670400    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tn7d\" (UniqueName: \"kubernetes.io/projected/9b01fd2c-9199-4288-b509-029da1d8b9a6-kube-api-access-8tn7d\") pod \"payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v\" (UID: \"9b01fd2c-9199-4288-b509-029da1d8b9a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"
Sep 17 03:37:08.670488 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.670442    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d11737d-b322-4bef-9c56-5d1bcb52d5cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls\" (UID: \"0d11737d-b322-4bef-9c56-5d1bcb52d5cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"
Sep 17 03:37:08.745895 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.745863    2557 generic.go:356] "Generic (PLEG): container finished" podID="72ba5071-edfb-4133-836b-531abf633199" containerID="6214dfb9284e0dc7a9ebde4b39b42ca1a4ed1ddfacdd9a4cfc3db2e0bc156e72" exitCode=0
Sep 17 03:37:08.746078 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.745909    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h" event={"ID":"72ba5071-edfb-4133-836b-531abf633199","Type":"ContainerDied","Data":"6214dfb9284e0dc7a9ebde4b39b42ca1a4ed1ddfacdd9a4cfc3db2e0bc156e72"}
Sep 17 03:37:08.746078 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.745957    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h" event={"ID":"72ba5071-edfb-4133-836b-531abf633199","Type":"ContainerDied","Data":"83074b9503da20f8c6e9e569692dd5f017ef71bbe6e12545ad8db0021ac860c7"}
Sep 17 03:37:08.746078 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.745914    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"
Sep 17 03:37:08.746078 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.745986    2557 scope.go:117] "RemoveContainer" containerID="6214dfb9284e0dc7a9ebde4b39b42ca1a4ed1ddfacdd9a4cfc3db2e0bc156e72"
Sep 17 03:37:08.756174 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.756153    2557 scope.go:117] "RemoveContainer" containerID="bccace71ee29b4672ff9ac4a7c6609b31a3edfff53eadec8aa5e767bada85427"
Sep 17 03:37:08.764922 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.764899    2557 scope.go:117] "RemoveContainer" containerID="6214dfb9284e0dc7a9ebde4b39b42ca1a4ed1ddfacdd9a4cfc3db2e0bc156e72"
Sep 17 03:37:08.765299 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:37:08.765267    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6214dfb9284e0dc7a9ebde4b39b42ca1a4ed1ddfacdd9a4cfc3db2e0bc156e72\": container with ID starting with 6214dfb9284e0dc7a9ebde4b39b42ca1a4ed1ddfacdd9a4cfc3db2e0bc156e72 not found: ID does not exist" containerID="6214dfb9284e0dc7a9ebde4b39b42ca1a4ed1ddfacdd9a4cfc3db2e0bc156e72"
Sep 17 03:37:08.765403 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.765296    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6214dfb9284e0dc7a9ebde4b39b42ca1a4ed1ddfacdd9a4cfc3db2e0bc156e72"} err="failed to get container status \"6214dfb9284e0dc7a9ebde4b39b42ca1a4ed1ddfacdd9a4cfc3db2e0bc156e72\": rpc error: code = NotFound desc = could not find container \"6214dfb9284e0dc7a9ebde4b39b42ca1a4ed1ddfacdd9a4cfc3db2e0bc156e72\": container with ID starting with 6214dfb9284e0dc7a9ebde4b39b42ca1a4ed1ddfacdd9a4cfc3db2e0bc156e72 not found: ID does not exist"
Sep 17 03:37:08.765403 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.765314    2557 scope.go:117] "RemoveContainer" containerID="bccace71ee29b4672ff9ac4a7c6609b31a3edfff53eadec8aa5e767bada85427"
Sep 17 03:37:08.765630 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:37:08.765609    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bccace71ee29b4672ff9ac4a7c6609b31a3edfff53eadec8aa5e767bada85427\": container with ID starting with bccace71ee29b4672ff9ac4a7c6609b31a3edfff53eadec8aa5e767bada85427 not found: ID does not exist" containerID="bccace71ee29b4672ff9ac4a7c6609b31a3edfff53eadec8aa5e767bada85427"
Sep 17 03:37:08.765669 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.765638    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bccace71ee29b4672ff9ac4a7c6609b31a3edfff53eadec8aa5e767bada85427"} err="failed to get container status \"bccace71ee29b4672ff9ac4a7c6609b31a3edfff53eadec8aa5e767bada85427\": rpc error: code = NotFound desc = could not find container \"bccace71ee29b4672ff9ac4a7c6609b31a3edfff53eadec8aa5e767bada85427\": container with ID starting with bccace71ee29b4672ff9ac4a7c6609b31a3edfff53eadec8aa5e767bada85427 not found: ID does not exist"
Sep 17 03:37:08.767354 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.767333    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"]
Sep 17 03:37:08.770682 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.770660    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-5062bf-openshift-default-6b9c9b9c5b-fp59h"]
Sep 17 03:37:08.771098 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.771079    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tn7d\" (UniqueName: \"kubernetes.io/projected/9b01fd2c-9199-4288-b509-029da1d8b9a6-kube-api-access-8tn7d\") pod \"payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v\" (UID: \"9b01fd2c-9199-4288-b509-029da1d8b9a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"
Sep 17 03:37:08.771142 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.771118    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d11737d-b322-4bef-9c56-5d1bcb52d5cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls\" (UID: \"0d11737d-b322-4bef-9c56-5d1bcb52d5cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"
Sep 17 03:37:08.771244 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.771231    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b01fd2c-9199-4288-b509-029da1d8b9a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v\" (UID: \"9b01fd2c-9199-4288-b509-029da1d8b9a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"
Sep 17 03:37:08.771288 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.771260    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf5rs\" (UniqueName: \"kubernetes.io/projected/0d11737d-b322-4bef-9c56-5d1bcb52d5cd-kube-api-access-qf5rs\") pod \"payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls\" (UID: \"0d11737d-b322-4bef-9c56-5d1bcb52d5cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"
Sep 17 03:37:08.771815 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.771799    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d11737d-b322-4bef-9c56-5d1bcb52d5cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls\" (UID: \"0d11737d-b322-4bef-9c56-5d1bcb52d5cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"
Sep 17 03:37:08.771899 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.771883    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b01fd2c-9199-4288-b509-029da1d8b9a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v\" (UID: \"9b01fd2c-9199-4288-b509-029da1d8b9a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"
Sep 17 03:37:08.778047 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.778020    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf5rs\" (UniqueName: \"kubernetes.io/projected/0d11737d-b322-4bef-9c56-5d1bcb52d5cd-kube-api-access-qf5rs\") pod \"payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls\" (UID: \"0d11737d-b322-4bef-9c56-5d1bcb52d5cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"
Sep 17 03:37:08.778129 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.778051    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tn7d\" (UniqueName: \"kubernetes.io/projected/9b01fd2c-9199-4288-b509-029da1d8b9a6-kube-api-access-8tn7d\") pod \"payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v\" (UID: \"9b01fd2c-9199-4288-b509-029da1d8b9a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"
Sep 17 03:37:08.895128 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.895105    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"
Sep 17 03:37:08.935686 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:08.935663    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"
Sep 17 03:37:09.172306 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:09.172279    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"]
Sep 17 03:37:09.174311 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:37:09.174287    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d11737d_b322_4bef_9c56_5d1bcb52d5cd.slice/crio-6f5de775a879cb6e88d6e25a7e6a0ce818580a67031b36e7e8ebdd28a0656aec WatchSource:0}: Error finding container 6f5de775a879cb6e88d6e25a7e6a0ce818580a67031b36e7e8ebdd28a0656aec: Status 404 returned error can't find the container with id 6f5de775a879cb6e88d6e25a7e6a0ce818580a67031b36e7e8ebdd28a0656aec
Sep 17 03:37:09.221047 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:09.221020    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"]
Sep 17 03:37:09.222975 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:37:09.222951    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b01fd2c_9199_4288_b509_029da1d8b9a6.slice/crio-8ead26145772d49dc0183eb8f7cab34a0a36918419a4c398d74b80635b33534c WatchSource:0}: Error finding container 8ead26145772d49dc0183eb8f7cab34a0a36918419a4c398d74b80635b33534c: Status 404 returned error can't find the container with id 8ead26145772d49dc0183eb8f7cab34a0a36918419a4c398d74b80635b33534c
Sep 17 03:37:09.754942 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:09.754809    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v" event={"ID":"9b01fd2c-9199-4288-b509-029da1d8b9a6","Type":"ContainerStarted","Data":"088b2e147b311fe7f42f953940e30915a6e9fffd7ff9ea8696d92bd7b88c3e8e"}
Sep 17 03:37:09.754942 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:09.754874    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v" event={"ID":"9b01fd2c-9199-4288-b509-029da1d8b9a6","Type":"ContainerStarted","Data":"8ead26145772d49dc0183eb8f7cab34a0a36918419a4c398d74b80635b33534c"}
Sep 17 03:37:09.754942 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:09.754904    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"
Sep 17 03:37:09.757841 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:09.757811    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls" event={"ID":"0d11737d-b322-4bef-9c56-5d1bcb52d5cd","Type":"ContainerStarted","Data":"1cd2e383319b34d5d2b602320f3a9185cdca269f7f9ff2c14ac5182859ba4736"}
Sep 17 03:37:09.758001 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:09.757878    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"
Sep 17 03:37:09.758001 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:09.757899    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls" event={"ID":"0d11737d-b322-4bef-9c56-5d1bcb52d5cd","Type":"ContainerStarted","Data":"6f5de775a879cb6e88d6e25a7e6a0ce818580a67031b36e7e8ebdd28a0656aec"}
Sep 17 03:37:09.772413 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:09.772359    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v" podStartSLOduration=1.772343984 podStartE2EDuration="1.772343984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:37:09.769289692 +0000 UTC m=+1080.372385916" watchObservedRunningTime="2026-09-17 03:37:09.772343984 +0000 UTC m=+1080.375439988"
Sep 17 03:37:09.793424 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:09.793371    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls" podStartSLOduration=1.79335216 podStartE2EDuration="1.79335216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:37:09.784087179 +0000 UTC m=+1080.387183183" watchObservedRunningTime="2026-09-17 03:37:09.79335216 +0000 UTC m=+1080.396448164"
Sep 17 03:37:09.918577 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:09.918546    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72ba5071-edfb-4133-836b-531abf633199" path="/var/lib/kubelet/pods/72ba5071-edfb-4133-836b-531abf633199/volumes"
Sep 17 03:37:11.095872 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:11.095840    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:11.096372 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:11.095881    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:11.108029 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:11.108007    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:11.782335 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:11.782305    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:37:20.766778 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:20.766745    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"
Sep 17 03:37:20.767170 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:20.766913    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"
Sep 17 03:37:35.433992 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.433958    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"]
Sep 17 03:37:35.438389 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.438361    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.440654 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.440625    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-fc820e-openshift-default-dockercfg-ldbjq\""
Sep 17 03:37:35.450122 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.450095    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"]
Sep 17 03:37:35.530219 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.530028    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-workload-socket\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.530219 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.530079    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-istio-data\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.530219 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.530124    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-istio-envoy\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.531015 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.530665    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-workload-certs\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.531015 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.530703    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4122da61-e79e-434b-83df-15053d1edeb5-istio-token\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.531015 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.530786    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4122da61-e79e-434b-83df-15053d1edeb5-istiod-ca-cert\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.531015 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.530813    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4122da61-e79e-434b-83df-15053d1edeb5-istio-podinfo\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.531015 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.530854    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-credential-socket\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.531015 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.530885    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dhq2\" (UniqueName: \"kubernetes.io/projected/4122da61-e79e-434b-83df-15053d1edeb5-kube-api-access-7dhq2\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.632731 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.632696    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-workload-certs\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.632928 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.632744    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4122da61-e79e-434b-83df-15053d1edeb5-istio-token\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.632928 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.632838    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4122da61-e79e-434b-83df-15053d1edeb5-istiod-ca-cert\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.632928 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.632865    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4122da61-e79e-434b-83df-15053d1edeb5-istio-podinfo\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.632928 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.632907    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-credential-socket\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.633145 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.633120    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-workload-certs\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.633241 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.633176    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dhq2\" (UniqueName: \"kubernetes.io/projected/4122da61-e79e-434b-83df-15053d1edeb5-kube-api-access-7dhq2\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.633314 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.633234    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-credential-socket\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.633314 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.633266    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-workload-socket\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.633314 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.633291    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-istio-data\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.633472 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.633343    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-istio-envoy\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.633681 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.633636    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-workload-socket\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.633819 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.633708    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-istio-data\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.634094 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.634047    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4122da61-e79e-434b-83df-15053d1edeb5-istiod-ca-cert\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.636363 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.636320    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-istio-envoy\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.636507 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.636491    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4122da61-e79e-434b-83df-15053d1edeb5-istio-podinfo\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.641029 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.641005    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4122da61-e79e-434b-83df-15053d1edeb5-istio-token\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.641324 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.641304    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dhq2\" (UniqueName: \"kubernetes.io/projected/4122da61-e79e-434b-83df-15053d1edeb5-kube-api-access-7dhq2\") pod \"e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") " pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.751950 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.751848    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:35.759463 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.759428    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 03:37:35.759583 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:35.759517    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm" podUID="4a823a95-f61d-40c9-aaca-238000c3e77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 03:37:36.035567 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:36.035535    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"]
Sep 17 03:37:36.037283 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:37:36.037242    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4122da61_e79e_434b_83df_15053d1edeb5.slice/crio-54eb8e29663b7802fdee5e1cde71efcce4bbe0e12aad5b817e9b53e2c29c2213 WatchSource:0}: Error finding container 54eb8e29663b7802fdee5e1cde71efcce4bbe0e12aad5b817e9b53e2c29c2213: Status 404 returned error can't find the container with id 54eb8e29663b7802fdee5e1cde71efcce4bbe0e12aad5b817e9b53e2c29c2213
Sep 17 03:37:36.039767 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:36.039732    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:37:36.039845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:36.039809    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:37:36.039892 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:36.039843    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:37:36.919682 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:36.919646    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n" event={"ID":"4122da61-e79e-434b-83df-15053d1edeb5","Type":"ContainerStarted","Data":"a0eb1fb74bb2642657d3339d887b15ca0033e6928068492ccfa2ba9e56c79e0a"}
Sep 17 03:37:36.920106 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:36.919697    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n" event={"ID":"4122da61-e79e-434b-83df-15053d1edeb5","Type":"ContainerStarted","Data":"54eb8e29663b7802fdee5e1cde71efcce4bbe0e12aad5b817e9b53e2c29c2213"}
Sep 17 03:37:36.939384 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:36.939340    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n" podStartSLOduration=1.939326098 podStartE2EDuration="1.939326098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:37:36.936414656 +0000 UTC m=+1107.539510651" watchObservedRunningTime="2026-09-17 03:37:36.939326098 +0000 UTC m=+1107.542422102"
Sep 17 03:37:37.752518 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:37.752484    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:37.758616 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:37.758593    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:37.927924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:37.927897    2557 generic.go:356] "Generic (PLEG): container finished" podID="4a823a95-f61d-40c9-aaca-238000c3e77e" containerID="5e31008dc9ce71df6f50cff759cb26eea9ca0e1d35612e0bc0ce6225b300a4bf" exitCode=0
Sep 17 03:37:37.928311 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:37.927975    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm" event={"ID":"4a823a95-f61d-40c9-aaca-238000c3e77e","Type":"ContainerDied","Data":"5e31008dc9ce71df6f50cff759cb26eea9ca0e1d35612e0bc0ce6225b300a4bf"}
Sep 17 03:37:37.928433 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:37.928411    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:37.928857 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:37.928834    2557 scope.go:117] "RemoveContainer" containerID="5e31008dc9ce71df6f50cff759cb26eea9ca0e1d35612e0bc0ce6225b300a4bf"
Sep 17 03:37:37.929753 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:37.929518    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:37:37.930126 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:37.930098    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:37:37.930255 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:37.930180    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:37:37.930328 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:37.930252    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:37:38.940179 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:38.940131    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm" event={"ID":"4a823a95-f61d-40c9-aaca-238000c3e77e","Type":"ContainerStarted","Data":"ca1182adc906a4aa5c29a20d431b591b7399c2c35f300a23737117b2874193e9"}
Sep 17 03:37:39.738254 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:39.738213    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:39.738456 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:39.738345    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 17 03:37:39.738456 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:39.738404    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm" podUID="4a823a95-f61d-40c9-aaca-238000c3e77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 17 03:37:40.738468 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:40.738425    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 17 03:37:40.738961 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:40.738512    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm" podUID="4a823a95-f61d-40c9-aaca-238000c3e77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 17 03:37:41.738899 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:41.737830    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 17 03:37:41.738899 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:41.737906    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm" podUID="4a823a95-f61d-40c9-aaca-238000c3e77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 17 03:37:42.741867 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:42.741830    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 17 03:37:42.742551 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:42.742515    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm" podUID="4a823a95-f61d-40c9-aaca-238000c3e77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 17 03:37:43.737968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:43.737929    2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Sep 17 03:37:43.738169 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:43.737998    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm" podUID="4a823a95-f61d-40c9-aaca-238000c3e77e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Sep 17 03:37:44.741663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:44.741633    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:44.742139 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:44.742051    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:37:44.742869 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:37:44.742851    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:38:04.786506 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:04.786468    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"]
Sep 17 03:38:04.787012 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:04.786725    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk" podUID="9cc5af42-e9d1-4741-89c1-5b742a035b4d" containerName="payload-pre-processing" containerID="cri-o://69ca6406c45efa50a2ab102492d52b5cf5bf4c50c9bfb8444c8a8e6eb3dc4b50" gracePeriod=30
Sep 17 03:38:05.118172 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:05.118132    2557 generic.go:356] "Generic (PLEG): container finished" podID="9cc5af42-e9d1-4741-89c1-5b742a035b4d" containerID="69ca6406c45efa50a2ab102492d52b5cf5bf4c50c9bfb8444c8a8e6eb3dc4b50" exitCode=0
Sep 17 03:38:05.118385 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:05.118257    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk" event={"ID":"9cc5af42-e9d1-4741-89c1-5b742a035b4d","Type":"ContainerDied","Data":"69ca6406c45efa50a2ab102492d52b5cf5bf4c50c9bfb8444c8a8e6eb3dc4b50"}
Sep 17 03:38:05.118385 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:05.118310    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk" event={"ID":"9cc5af42-e9d1-4741-89c1-5b742a035b4d","Type":"ContainerDied","Data":"7e4469d2d4e09f1340505bb29d33a1fb49b2ec71985b45f0bfb1e0a9aa188e1f"}
Sep 17 03:38:05.118385 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:05.118332    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e4469d2d4e09f1340505bb29d33a1fb49b2ec71985b45f0bfb1e0a9aa188e1f"
Sep 17 03:38:05.122809 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:05.122784    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"
Sep 17 03:38:05.268716 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:05.268683    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcx4p\" (UniqueName: \"kubernetes.io/projected/9cc5af42-e9d1-4741-89c1-5b742a035b4d-kube-api-access-pcx4p\") pod \"9cc5af42-e9d1-4741-89c1-5b742a035b4d\" (UID: \"9cc5af42-e9d1-4741-89c1-5b742a035b4d\") "
Sep 17 03:38:05.268918 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:05.268848    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc5af42-e9d1-4741-89c1-5b742a035b4d-plugins-config-volume\") pod \"9cc5af42-e9d1-4741-89c1-5b742a035b4d\" (UID: \"9cc5af42-e9d1-4741-89c1-5b742a035b4d\") "
Sep 17 03:38:05.269222 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:05.269184    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc5af42-e9d1-4741-89c1-5b742a035b4d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9cc5af42-e9d1-4741-89c1-5b742a035b4d" (UID: "9cc5af42-e9d1-4741-89c1-5b742a035b4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:38:05.270933 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:05.270911    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc5af42-e9d1-4741-89c1-5b742a035b4d-kube-api-access-pcx4p" (OuterVolumeSpecName: "kube-api-access-pcx4p") pod "9cc5af42-e9d1-4741-89c1-5b742a035b4d" (UID: "9cc5af42-e9d1-4741-89c1-5b742a035b4d"). InnerVolumeSpecName "kube-api-access-pcx4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:38:05.370702 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:05.370670    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc5af42-e9d1-4741-89c1-5b742a035b4d-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:38:05.370875 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:05.370706    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcx4p\" (UniqueName: \"kubernetes.io/projected/9cc5af42-e9d1-4741-89c1-5b742a035b4d-kube-api-access-pcx4p\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:38:06.126479 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:06.126443    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"
Sep 17 03:38:06.144534 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:06.144498    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"]
Sep 17 03:38:06.148755 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:06.148724    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-297324-68754dd758-d7vzk"]
Sep 17 03:38:07.916365 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:07.916329    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cc5af42-e9d1-4741-89c1-5b742a035b4d" path="/var/lib/kubelet/pods/9cc5af42-e9d1-4741-89c1-5b742a035b4d/volumes"
Sep 17 03:38:10.065575 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:10.065544    2557 scope.go:117] "RemoveContainer" containerID="69ca6406c45efa50a2ab102492d52b5cf5bf4c50c9bfb8444c8a8e6eb3dc4b50"
Sep 17 03:38:11.186879 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:11.186845    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"]
Sep 17 03:38:11.187372 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:11.187095    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz" podUID="648fc366-4335-4342-9d41-81e446da0dd2" containerName="istio-proxy" containerID="cri-o://7bc7bd3a33071bfc2ddbe96a1c6d2f8c2d38c15fb82c09488e096f38200d8a34" gracePeriod=30
Sep 17 03:38:13.191391 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:13.191349    2557 patch_prober.go:28] interesting pod/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 03:38:13.191844 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:13.191421    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz" podUID="648fc366-4335-4342-9d41-81e446da0dd2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 03:38:14.184913 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:14.184883    2557 generic.go:356] "Generic (PLEG): container finished" podID="4122da61-e79e-434b-83df-15053d1edeb5" containerID="a0eb1fb74bb2642657d3339d887b15ca0033e6928068492ccfa2ba9e56c79e0a" exitCode=0
Sep 17 03:38:14.185108 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:14.184952    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n" event={"ID":"4122da61-e79e-434b-83df-15053d1edeb5","Type":"ContainerDied","Data":"a0eb1fb74bb2642657d3339d887b15ca0033e6928068492ccfa2ba9e56c79e0a"}
Sep 17 03:38:14.185408 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:14.185393    2557 scope.go:117] "RemoveContainer" containerID="a0eb1fb74bb2642657d3339d887b15ca0033e6928068492ccfa2ba9e56c79e0a"
Sep 17 03:38:14.186610 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:14.186583    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:38:14.186673 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:14.186651    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:38:14.186711 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:14.186686    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:38:15.193319 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:15.193285    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n" event={"ID":"4122da61-e79e-434b-83df-15053d1edeb5","Type":"ContainerStarted","Data":"4ba83ac674dae741d583ef45ac5d2b140a33c52b738310a55c0e37fcd42a5422"}
Sep 17 03:38:15.752300 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:15.752261    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:38:15.752577 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:15.752552    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 17 03:38:15.752660 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:15.752626    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n" podUID="4122da61-e79e-434b-83df-15053d1edeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 17 03:38:16.487511 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.487485    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:38:16.607052 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607020    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-istio-envoy\") pod \"648fc366-4335-4342-9d41-81e446da0dd2\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") "
Sep 17 03:38:16.607234 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607075    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-workload-socket\") pod \"648fc366-4335-4342-9d41-81e446da0dd2\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") "
Sep 17 03:38:16.607234 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607103    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648fc366-4335-4342-9d41-81e446da0dd2-istio-podinfo\") pod \"648fc366-4335-4342-9d41-81e446da0dd2\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") "
Sep 17 03:38:16.607234 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607146    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-workload-certs\") pod \"648fc366-4335-4342-9d41-81e446da0dd2\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") "
Sep 17 03:38:16.607234 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607184    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648fc366-4335-4342-9d41-81e446da0dd2-istiod-ca-cert\") pod \"648fc366-4335-4342-9d41-81e446da0dd2\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") "
Sep 17 03:38:16.607470 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607258    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648fc366-4335-4342-9d41-81e446da0dd2-istio-token\") pod \"648fc366-4335-4342-9d41-81e446da0dd2\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") "
Sep 17 03:38:16.607470 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607282    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-istio-data\") pod \"648fc366-4335-4342-9d41-81e446da0dd2\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") "
Sep 17 03:38:16.607470 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607310    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzp8k\" (UniqueName: \"kubernetes.io/projected/648fc366-4335-4342-9d41-81e446da0dd2-kube-api-access-tzp8k\") pod \"648fc366-4335-4342-9d41-81e446da0dd2\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") "
Sep 17 03:38:16.607470 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607350    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-credential-socket\") pod \"648fc366-4335-4342-9d41-81e446da0dd2\" (UID: \"648fc366-4335-4342-9d41-81e446da0dd2\") "
Sep 17 03:38:16.607470 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607407    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "648fc366-4335-4342-9d41-81e446da0dd2" (UID: "648fc366-4335-4342-9d41-81e446da0dd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:38:16.607736 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607476    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "648fc366-4335-4342-9d41-81e446da0dd2" (UID: "648fc366-4335-4342-9d41-81e446da0dd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:38:16.607797 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607769    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "648fc366-4335-4342-9d41-81e446da0dd2" (UID: "648fc366-4335-4342-9d41-81e446da0dd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:38:16.607873 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607852    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:38:16.607942 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607879    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:38:16.607942 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607893    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-istio-data" (OuterVolumeSpecName: "istio-data") pod "648fc366-4335-4342-9d41-81e446da0dd2" (UID: "648fc366-4335-4342-9d41-81e446da0dd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:38:16.608052 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.607998    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648fc366-4335-4342-9d41-81e446da0dd2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "648fc366-4335-4342-9d41-81e446da0dd2" (UID: "648fc366-4335-4342-9d41-81e446da0dd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:38:16.609819 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.609795    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648fc366-4335-4342-9d41-81e446da0dd2-kube-api-access-tzp8k" (OuterVolumeSpecName: "kube-api-access-tzp8k") pod "648fc366-4335-4342-9d41-81e446da0dd2" (UID: "648fc366-4335-4342-9d41-81e446da0dd2"). InnerVolumeSpecName "kube-api-access-tzp8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:38:16.609975 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.609954    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "648fc366-4335-4342-9d41-81e446da0dd2" (UID: "648fc366-4335-4342-9d41-81e446da0dd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:38:16.610035 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.609988    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/648fc366-4335-4342-9d41-81e446da0dd2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "648fc366-4335-4342-9d41-81e446da0dd2" (UID: "648fc366-4335-4342-9d41-81e446da0dd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:38:16.610073 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.610038    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648fc366-4335-4342-9d41-81e446da0dd2-istio-token" (OuterVolumeSpecName: "istio-token") pod "648fc366-4335-4342-9d41-81e446da0dd2" (UID: "648fc366-4335-4342-9d41-81e446da0dd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:38:16.709084 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.709057    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:38:16.709237 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.709103    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/648fc366-4335-4342-9d41-81e446da0dd2-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:38:16.709237 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.709118    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/648fc366-4335-4342-9d41-81e446da0dd2-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:38:16.709237 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.709132    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/648fc366-4335-4342-9d41-81e446da0dd2-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:38:16.709237 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.709145    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:38:16.709237 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.709177    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzp8k\" (UniqueName: \"kubernetes.io/projected/648fc366-4335-4342-9d41-81e446da0dd2-kube-api-access-tzp8k\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:38:16.709237 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.709219    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/648fc366-4335-4342-9d41-81e446da0dd2-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:38:16.752706 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.752619    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 17 03:38:16.752706 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:16.752681    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n" podUID="4122da61-e79e-434b-83df-15053d1edeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 17 03:38:17.206941 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:17.206904    2557 generic.go:356] "Generic (PLEG): container finished" podID="648fc366-4335-4342-9d41-81e446da0dd2" containerID="7bc7bd3a33071bfc2ddbe96a1c6d2f8c2d38c15fb82c09488e096f38200d8a34" exitCode=0
Sep 17 03:38:17.207110 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:17.206945    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"
Sep 17 03:38:17.207110 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:17.207033    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz" event={"ID":"648fc366-4335-4342-9d41-81e446da0dd2","Type":"ContainerDied","Data":"7bc7bd3a33071bfc2ddbe96a1c6d2f8c2d38c15fb82c09488e096f38200d8a34"}
Sep 17 03:38:17.207110 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:17.207079    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz" event={"ID":"648fc366-4335-4342-9d41-81e446da0dd2","Type":"ContainerDied","Data":"f0c2c0d7be763d296ece8b7501c3c7eb1bfa506244a467206ac4d7924a2808a8"}
Sep 17 03:38:17.207110 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:17.207101    2557 scope.go:117] "RemoveContainer" containerID="7bc7bd3a33071bfc2ddbe96a1c6d2f8c2d38c15fb82c09488e096f38200d8a34"
Sep 17 03:38:17.217261 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:17.217245    2557 scope.go:117] "RemoveContainer" containerID="175953a3d6fbd8b092fa5bfa0ced93fbffcab2ab1c362255695492ddddb3aaab"
Sep 17 03:38:17.231390 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:17.231322    2557 scope.go:117] "RemoveContainer" containerID="7bc7bd3a33071bfc2ddbe96a1c6d2f8c2d38c15fb82c09488e096f38200d8a34"
Sep 17 03:38:17.231721 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:38:17.231690    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc7bd3a33071bfc2ddbe96a1c6d2f8c2d38c15fb82c09488e096f38200d8a34\": container with ID starting with 7bc7bd3a33071bfc2ddbe96a1c6d2f8c2d38c15fb82c09488e096f38200d8a34 not found: ID does not exist" containerID="7bc7bd3a33071bfc2ddbe96a1c6d2f8c2d38c15fb82c09488e096f38200d8a34"
Sep 17 03:38:17.231825 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:17.231731    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc7bd3a33071bfc2ddbe96a1c6d2f8c2d38c15fb82c09488e096f38200d8a34"} err="failed to get container status \"7bc7bd3a33071bfc2ddbe96a1c6d2f8c2d38c15fb82c09488e096f38200d8a34\": rpc error: code = NotFound desc = could not find container \"7bc7bd3a33071bfc2ddbe96a1c6d2f8c2d38c15fb82c09488e096f38200d8a34\": container with ID starting with 7bc7bd3a33071bfc2ddbe96a1c6d2f8c2d38c15fb82c09488e096f38200d8a34 not found: ID does not exist"
Sep 17 03:38:17.231825 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:17.231753    2557 scope.go:117] "RemoveContainer" containerID="175953a3d6fbd8b092fa5bfa0ced93fbffcab2ab1c362255695492ddddb3aaab"
Sep 17 03:38:17.232035 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:38:17.232007    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175953a3d6fbd8b092fa5bfa0ced93fbffcab2ab1c362255695492ddddb3aaab\": container with ID starting with 175953a3d6fbd8b092fa5bfa0ced93fbffcab2ab1c362255695492ddddb3aaab not found: ID does not exist" containerID="175953a3d6fbd8b092fa5bfa0ced93fbffcab2ab1c362255695492ddddb3aaab"
Sep 17 03:38:17.232116 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:17.232043    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175953a3d6fbd8b092fa5bfa0ced93fbffcab2ab1c362255695492ddddb3aaab"} err="failed to get container status \"175953a3d6fbd8b092fa5bfa0ced93fbffcab2ab1c362255695492ddddb3aaab\": rpc error: code = NotFound desc = could not find container \"175953a3d6fbd8b092fa5bfa0ced93fbffcab2ab1c362255695492ddddb3aaab\": container with ID starting with 175953a3d6fbd8b092fa5bfa0ced93fbffcab2ab1c362255695492ddddb3aaab not found: ID does not exist"
Sep 17 03:38:17.236240 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:17.236187    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"]
Sep 17 03:38:17.236339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:17.236263    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-297324-openshift-default-5fd877d844-g49lz"]
Sep 17 03:38:17.752915 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:17.752869    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 17 03:38:17.753422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:17.752941    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n" podUID="4122da61-e79e-434b-83df-15053d1edeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 17 03:38:17.915975 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:17.915940    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="648fc366-4335-4342-9d41-81e446da0dd2" path="/var/lib/kubelet/pods/648fc366-4335-4342-9d41-81e446da0dd2/volumes"
Sep 17 03:38:18.752667 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:18.752625    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 17 03:38:18.752864 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:18.752686    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n" podUID="4122da61-e79e-434b-83df-15053d1edeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 17 03:38:19.752436 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:19.752400    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 17 03:38:19.752860 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:19.752458    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n" podUID="4122da61-e79e-434b-83df-15053d1edeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 17 03:38:20.752712 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:20.752679    2557 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 17 03:38:20.753095 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:20.752740    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n" podUID="4122da61-e79e-434b-83df-15053d1edeb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 17 03:38:21.204966 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.204932    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"]
Sep 17 03:38:21.205683 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.205666    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="648fc366-4335-4342-9d41-81e446da0dd2" containerName="istio-proxy"
Sep 17 03:38:21.205740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.205687    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="648fc366-4335-4342-9d41-81e446da0dd2" containerName="istio-proxy"
Sep 17 03:38:21.205740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.205706    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cc5af42-e9d1-4741-89c1-5b742a035b4d" containerName="payload-pre-processing"
Sep 17 03:38:21.205740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.205715    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cc5af42-e9d1-4741-89c1-5b742a035b4d" containerName="payload-pre-processing"
Sep 17 03:38:21.205842 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.205825    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="648fc366-4335-4342-9d41-81e446da0dd2" containerName="istio-proxy"
Sep 17 03:38:21.205877 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.205843    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cc5af42-e9d1-4741-89c1-5b742a035b4d" containerName="payload-pre-processing"
Sep 17 03:38:21.211311 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.211283    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:21.213284 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.213255    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c10ab8fb-serving-cert\""
Sep 17 03:38:21.219593 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.219570    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"]
Sep 17 03:38:21.264622 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.264585    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f3f719-8733-4c14-8d24-217810ed59a3-openshift-service-ca\") pod \"maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:21.264764 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.264623    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkqr9\" (UniqueName: \"kubernetes.io/projected/55f3f719-8733-4c14-8d24-217810ed59a3-kube-api-access-zkqr9\") pod \"maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:21.264764 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.264736    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-maas-api-tls\") pod \"maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:21.264853 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.264788    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-metrics-certs\") pod \"maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:21.365817 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.365784    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f3f719-8733-4c14-8d24-217810ed59a3-openshift-service-ca\") pod \"maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:21.365817 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.365815    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqr9\" (UniqueName: \"kubernetes.io/projected/55f3f719-8733-4c14-8d24-217810ed59a3-kube-api-access-zkqr9\") pod \"maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:21.366053 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.365858    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-maas-api-tls\") pod \"maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:21.366053 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.365881    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-metrics-certs\") pod \"maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:21.366053 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:38:21.366015    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-serving-cert: secret "maas-api-e2e-mt-c10ab8fb-serving-cert" not found
Sep 17 03:38:21.366234 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:38:21.366092    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-maas-api-tls podName:55f3f719-8733-4c14-8d24-217810ed59a3 nodeName:}" failed. No retries permitted until 2026-09-17 03:38:21.86607607 +0000 UTC m=+1152.469172051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-maas-api-tls") pod "maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh" (UID: "55f3f719-8733-4c14-8d24-217810ed59a3") : secret "maas-api-e2e-mt-c10ab8fb-serving-cert" not found
Sep 17 03:38:21.366602 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.366581    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f3f719-8733-4c14-8d24-217810ed59a3-openshift-service-ca\") pod \"maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:21.368223 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.368186    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-metrics-certs\") pod \"maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:21.374256 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.374230    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkqr9\" (UniqueName: \"kubernetes.io/projected/55f3f719-8733-4c14-8d24-217810ed59a3-kube-api-access-zkqr9\") pod \"maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:21.756666 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.756640    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:38:21.757111 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.756910    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:38:21.757528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.757512    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:38:21.784003 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.783978    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"]
Sep 17 03:38:21.784621 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.784598    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="648fc366-4335-4342-9d41-81e446da0dd2" containerName="istio-proxy"
Sep 17 03:38:21.784621 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.784618    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="648fc366-4335-4342-9d41-81e446da0dd2" containerName="istio-proxy"
Sep 17 03:38:21.784810 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.784726    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="648fc366-4335-4342-9d41-81e446da0dd2" containerName="istio-proxy"
Sep 17 03:38:21.788368 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.788348    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"
Sep 17 03:38:21.790841 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.790819    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c10ab8fb\""
Sep 17 03:38:21.793798 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.793777    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"]
Sep 17 03:38:21.872236 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.872185    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2brpv\" (UniqueName: \"kubernetes.io/projected/f9f91d55-34b9-4bc0-a668-c59c64f2428a-kube-api-access-2brpv\") pod \"payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65\" (UID: \"f9f91d55-34b9-4bc0-a668-c59c64f2428a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"
Sep 17 03:38:21.872415 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.872291    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-maas-api-tls\") pod \"maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:21.872415 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.872409    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f91d55-34b9-4bc0-a668-c59c64f2428a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65\" (UID: \"f9f91d55-34b9-4bc0-a668-c59c64f2428a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"
Sep 17 03:38:21.872539 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:38:21.872427    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-serving-cert: secret "maas-api-e2e-mt-c10ab8fb-serving-cert" not found
Sep 17 03:38:21.872539 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:38:21.872493    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-maas-api-tls podName:55f3f719-8733-4c14-8d24-217810ed59a3 nodeName:}" failed. No retries permitted until 2026-09-17 03:38:22.872473749 +0000 UTC m=+1153.475569754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-maas-api-tls") pod "maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh" (UID: "55f3f719-8733-4c14-8d24-217810ed59a3") : secret "maas-api-e2e-mt-c10ab8fb-serving-cert" not found
Sep 17 03:38:21.974367 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.973515    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f91d55-34b9-4bc0-a668-c59c64f2428a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65\" (UID: \"f9f91d55-34b9-4bc0-a668-c59c64f2428a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"
Sep 17 03:38:21.974367 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.974145    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2brpv\" (UniqueName: \"kubernetes.io/projected/f9f91d55-34b9-4bc0-a668-c59c64f2428a-kube-api-access-2brpv\") pod \"payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65\" (UID: \"f9f91d55-34b9-4bc0-a668-c59c64f2428a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"
Sep 17 03:38:21.974672 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.974645    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f91d55-34b9-4bc0-a668-c59c64f2428a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65\" (UID: \"f9f91d55-34b9-4bc0-a668-c59c64f2428a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"
Sep 17 03:38:21.981614 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:21.981590    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2brpv\" (UniqueName: \"kubernetes.io/projected/f9f91d55-34b9-4bc0-a668-c59c64f2428a-kube-api-access-2brpv\") pod \"payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65\" (UID: \"f9f91d55-34b9-4bc0-a668-c59c64f2428a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"
Sep 17 03:38:22.102090 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:22.102015    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"
Sep 17 03:38:22.379378 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:22.379350    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"]
Sep 17 03:38:22.380658 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:38:22.380619    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f91d55_34b9_4bc0_a668_c59c64f2428a.slice/crio-f2c85e28d54c746f3630828dfdfbe3a4a55a5e00e88b309d772ca430762c6b5b WatchSource:0}: Error finding container f2c85e28d54c746f3630828dfdfbe3a4a55a5e00e88b309d772ca430762c6b5b: Status 404 returned error can't find the container with id f2c85e28d54c746f3630828dfdfbe3a4a55a5e00e88b309d772ca430762c6b5b
Sep 17 03:38:22.892431 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:22.892392    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-maas-api-tls\") pod \"maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:22.894953 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:22.894926    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-maas-api-tls\") pod \"maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:23.029339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:23.029305    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:23.251167 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:23.251087    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65" event={"ID":"f9f91d55-34b9-4bc0-a668-c59c64f2428a","Type":"ContainerStarted","Data":"43455e4a32f4d12766b08070fbfe1bd77a7961bc7ac9b177b2aa88fb7e2e19e4"}
Sep 17 03:38:23.251370 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:23.251171    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65" event={"ID":"f9f91d55-34b9-4bc0-a668-c59c64f2428a","Type":"ContainerStarted","Data":"f2c85e28d54c746f3630828dfdfbe3a4a55a5e00e88b309d772ca430762c6b5b"}
Sep 17 03:38:23.251370 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:23.251221    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"
Sep 17 03:38:23.266347 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:23.266303    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65" podStartSLOduration=2.2662876 podStartE2EDuration="2.2662876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:38:23.265398641 +0000 UTC m=+1153.868494646" watchObservedRunningTime="2026-09-17 03:38:23.2662876 +0000 UTC m=+1153.869383605"
Sep 17 03:38:23.304121 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:23.304097    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"]
Sep 17 03:38:23.305964 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:38:23.305938    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f3f719_8733_4c14_8d24_217810ed59a3.slice/crio-303ec81988825339afa538490fa4e12ac7520a55a5e4d6a3c97ad6c45761be69 WatchSource:0}: Error finding container 303ec81988825339afa538490fa4e12ac7520a55a5e4d6a3c97ad6c45761be69: Status 404 returned error can't find the container with id 303ec81988825339afa538490fa4e12ac7520a55a5e4d6a3c97ad6c45761be69
Sep 17 03:38:24.259013 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:24.258974    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh" event={"ID":"55f3f719-8733-4c14-8d24-217810ed59a3","Type":"ContainerStarted","Data":"5b63c262305e24ec4fd073e61772cc9e48e88f0cb5d224c62885da6d1a8211f5"}
Sep 17 03:38:24.259480 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:24.259040    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh" event={"ID":"55f3f719-8733-4c14-8d24-217810ed59a3","Type":"ContainerStarted","Data":"303ec81988825339afa538490fa4e12ac7520a55a5e4d6a3c97ad6c45761be69"}
Sep 17 03:38:24.276403 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:24.276354    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh" podStartSLOduration=2.917529811 podStartE2EDuration="3.276339716s" totalImagesPullingTime="358.809905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:38:24.274914061 +0000 UTC m=+1154.878010066" watchObservedRunningTime="2026-09-17 03:38:24.276339716 +0000 UTC m=+1154.879435721"
Sep 17 03:38:25.265726 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:25.265698    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:31.277055 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:31.277028    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:38:34.261739 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:38:34.261662    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"
Sep 17 03:39:09.937716 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:09.937686    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:39:09.940183 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:09.940163    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:39:10.086879 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:10.086854    2557 scope.go:117] "RemoveContainer" containerID="9f0eb36e95b521fd49ca18b66d7da9421db92ba841476b2f6f2bd39a93a78b23"
Sep 17 03:39:46.764539 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.764503    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"]
Sep 17 03:39:46.768561 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.768543    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"
Sep 17 03:39:46.770946 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.770924    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-f6d581-dockercfg-zg65j\""
Sep 17 03:39:46.771064 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.770963    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-f6d581\""
Sep 17 03:39:46.777529 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.777510    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"]
Sep 17 03:39:46.802367 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.801015    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"]
Sep 17 03:39:46.807531 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.806919    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"
Sep 17 03:39:46.814729 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.814707    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"]
Sep 17 03:39:46.854508 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.854479    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cqcc\" (UniqueName: \"kubernetes.io/projected/8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a-kube-api-access-9cqcc\") pod \"payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg\" (UID: \"8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"
Sep 17 03:39:46.854704 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.854646    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg\" (UID: \"8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"
Sep 17 03:39:46.854813 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.854789    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7tvr\" (UniqueName: \"kubernetes.io/projected/3e0dd334-39e0-4176-b826-fe45f297879f-kube-api-access-h7tvr\") pod \"payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx\" (UID: \"3e0dd334-39e0-4176-b826-fe45f297879f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"
Sep 17 03:39:46.855080 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.855041    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0dd334-39e0-4176-b826-fe45f297879f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx\" (UID: \"3e0dd334-39e0-4176-b826-fe45f297879f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"
Sep 17 03:39:46.957050 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.956971    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cqcc\" (UniqueName: \"kubernetes.io/projected/8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a-kube-api-access-9cqcc\") pod \"payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg\" (UID: \"8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"
Sep 17 03:39:46.957228 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.957094    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg\" (UID: \"8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"
Sep 17 03:39:46.957228 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.957181    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7tvr\" (UniqueName: \"kubernetes.io/projected/3e0dd334-39e0-4176-b826-fe45f297879f-kube-api-access-h7tvr\") pod \"payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx\" (UID: \"3e0dd334-39e0-4176-b826-fe45f297879f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"
Sep 17 03:39:46.957228 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.957219    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0dd334-39e0-4176-b826-fe45f297879f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx\" (UID: \"3e0dd334-39e0-4176-b826-fe45f297879f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"
Sep 17 03:39:46.957880 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.957864    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0dd334-39e0-4176-b826-fe45f297879f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx\" (UID: \"3e0dd334-39e0-4176-b826-fe45f297879f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"
Sep 17 03:39:46.958055 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.958033    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg\" (UID: \"8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"
Sep 17 03:39:46.965141 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.965110    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cqcc\" (UniqueName: \"kubernetes.io/projected/8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a-kube-api-access-9cqcc\") pod \"payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg\" (UID: \"8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"
Sep 17 03:39:46.965476 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:46.965458    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7tvr\" (UniqueName: \"kubernetes.io/projected/3e0dd334-39e0-4176-b826-fe45f297879f-kube-api-access-h7tvr\") pod \"payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx\" (UID: \"3e0dd334-39e0-4176-b826-fe45f297879f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"
Sep 17 03:39:47.081111 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:47.081048    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"
Sep 17 03:39:47.121458 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:47.120228    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"
Sep 17 03:39:47.367864 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:47.367842    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"]
Sep 17 03:39:47.371222 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:39:47.371167    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0dd334_39e0_4176_b826_fe45f297879f.slice/crio-8d978a2fd55f388d1ab663de55f91b25c36fb7d160ea71626434366dd6cc1dfd WatchSource:0}: Error finding container 8d978a2fd55f388d1ab663de55f91b25c36fb7d160ea71626434366dd6cc1dfd: Status 404 returned error can't find the container with id 8d978a2fd55f388d1ab663de55f91b25c36fb7d160ea71626434366dd6cc1dfd
Sep 17 03:39:47.419986 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:47.419961    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"]
Sep 17 03:39:47.423956 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:39:47.423923    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfe9c51_5ca0_47c7_ad6a_5a3aa274318a.slice/crio-3cef1230bfb8673cb21c2afe3b34580fbc27c4a739fc4ce34bdc560de1f9ddcd WatchSource:0}: Error finding container 3cef1230bfb8673cb21c2afe3b34580fbc27c4a739fc4ce34bdc560de1f9ddcd: Status 404 returned error can't find the container with id 3cef1230bfb8673cb21c2afe3b34580fbc27c4a739fc4ce34bdc560de1f9ddcd
Sep 17 03:39:47.765820 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:47.765737    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg" event={"ID":"8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a","Type":"ContainerStarted","Data":"b43e6299c18d035d966e09b887ec644d116c33b76678b8f4b7cd19342250a5d6"}
Sep 17 03:39:47.766373 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:47.766341    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg" event={"ID":"8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a","Type":"ContainerStarted","Data":"3cef1230bfb8673cb21c2afe3b34580fbc27c4a739fc4ce34bdc560de1f9ddcd"}
Sep 17 03:39:47.766506 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:47.766385    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"
Sep 17 03:39:47.767531 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:47.767508    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx" event={"ID":"3e0dd334-39e0-4176-b826-fe45f297879f","Type":"ContainerStarted","Data":"c525152048cd5b8b5c474b3796514e0eaafed5baae211499a076fae18eee0815"}
Sep 17 03:39:47.767641 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:47.767557    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"
Sep 17 03:39:47.767641 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:47.767581    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx" event={"ID":"3e0dd334-39e0-4176-b826-fe45f297879f","Type":"ContainerStarted","Data":"8d978a2fd55f388d1ab663de55f91b25c36fb7d160ea71626434366dd6cc1dfd"}
Sep 17 03:39:47.783035 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:47.782987    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg" podStartSLOduration=1.782973328 podStartE2EDuration="1.782973328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:39:47.780110774 +0000 UTC m=+1238.383206771" watchObservedRunningTime="2026-09-17 03:39:47.782973328 +0000 UTC m=+1238.386069334"
Sep 17 03:39:47.795350 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:47.795309    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx" podStartSLOduration=1.7952963259999999 podStartE2EDuration="1.795296326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:39:47.794124131 +0000 UTC m=+1238.397220139" watchObservedRunningTime="2026-09-17 03:39:47.795296326 +0000 UTC m=+1238.398392327"
Sep 17 03:39:58.775687 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:58.775655    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"
Sep 17 03:39:58.776099 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:39:58.775922    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"
Sep 17 03:40:10.114743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:10.114668    2557 scope.go:117] "RemoveContainer" containerID="dd91b0ad824d28f76a92b5c58901f3a913cd73d563f64b3adcd4c4de792c4adb"
Sep 17 03:40:10.124576 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:10.124544    2557 scope.go:117] "RemoveContainer" containerID="608ff78affd25e9abc5ab36ee3e3d8d3c5b430dd903b7097ec9896b037c96611"
Sep 17 03:40:20.199797 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:20.199763    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"]
Sep 17 03:40:35.224706 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.224641    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm" podUID="aca02225-cef7-4f9c-8545-6c07bf293a3e" containerName="main" containerID="cri-o://4d2f981bd5dc73f066efa6d1e18c9fe94078161324783065a6f61a1b5339a3e9" gracePeriod=45
Sep 17 03:40:35.478221 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.478156    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:40:35.494831 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.494805    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwrl4\" (UniqueName: \"kubernetes.io/projected/aca02225-cef7-4f9c-8545-6c07bf293a3e-kube-api-access-jwrl4\") pod \"aca02225-cef7-4f9c-8545-6c07bf293a3e\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") "
Sep 17 03:40:35.494997 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.494981    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-tmp-dir\") pod \"aca02225-cef7-4f9c-8545-6c07bf293a3e\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") "
Sep 17 03:40:35.495294 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.495025    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-dshm\") pod \"aca02225-cef7-4f9c-8545-6c07bf293a3e\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") "
Sep 17 03:40:35.495294 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.495136    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-model-cache\") pod \"aca02225-cef7-4f9c-8545-6c07bf293a3e\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") "
Sep 17 03:40:35.495294 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.495291    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca02225-cef7-4f9c-8545-6c07bf293a3e-tls-certs\") pod \"aca02225-cef7-4f9c-8545-6c07bf293a3e\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") "
Sep 17 03:40:35.495525 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.495319    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-home\") pod \"aca02225-cef7-4f9c-8545-6c07bf293a3e\" (UID: \"aca02225-cef7-4f9c-8545-6c07bf293a3e\") "
Sep 17 03:40:35.495955 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.495859    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-home" (OuterVolumeSpecName: "home") pod "aca02225-cef7-4f9c-8545-6c07bf293a3e" (UID: "aca02225-cef7-4f9c-8545-6c07bf293a3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:40:35.496082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.496011    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aca02225-cef7-4f9c-8545-6c07bf293a3e" (UID: "aca02225-cef7-4f9c-8545-6c07bf293a3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:40:35.496283 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.496240    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-model-cache" (OuterVolumeSpecName: "model-cache") pod "aca02225-cef7-4f9c-8545-6c07bf293a3e" (UID: "aca02225-cef7-4f9c-8545-6c07bf293a3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:40:35.498174 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.498148    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca02225-cef7-4f9c-8545-6c07bf293a3e-kube-api-access-jwrl4" (OuterVolumeSpecName: "kube-api-access-jwrl4") pod "aca02225-cef7-4f9c-8545-6c07bf293a3e" (UID: "aca02225-cef7-4f9c-8545-6c07bf293a3e"). InnerVolumeSpecName "kube-api-access-jwrl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:40:35.498409 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.498385    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-dshm" (OuterVolumeSpecName: "dshm") pod "aca02225-cef7-4f9c-8545-6c07bf293a3e" (UID: "aca02225-cef7-4f9c-8545-6c07bf293a3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:40:35.498560 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.498538    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aca02225-cef7-4f9c-8545-6c07bf293a3e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aca02225-cef7-4f9c-8545-6c07bf293a3e" (UID: "aca02225-cef7-4f9c-8545-6c07bf293a3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:40:35.597186 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.597155    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aca02225-cef7-4f9c-8545-6c07bf293a3e-tls-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:40:35.597186 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.597184    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-home\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:40:35.597373 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.597211    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwrl4\" (UniqueName: \"kubernetes.io/projected/aca02225-cef7-4f9c-8545-6c07bf293a3e-kube-api-access-jwrl4\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:40:35.597373 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.597225    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-tmp-dir\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:40:35.597373 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.597238    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-dshm\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:40:35.597373 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:35.597248    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aca02225-cef7-4f9c-8545-6c07bf293a3e-model-cache\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:40:36.047737 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:36.047703    2557 generic.go:356] "Generic (PLEG): container finished" podID="aca02225-cef7-4f9c-8545-6c07bf293a3e" containerID="4d2f981bd5dc73f066efa6d1e18c9fe94078161324783065a6f61a1b5339a3e9" exitCode=0
Sep 17 03:40:36.047737 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:36.047736    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"
Sep 17 03:40:36.047932 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:36.047790    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm" event={"ID":"aca02225-cef7-4f9c-8545-6c07bf293a3e","Type":"ContainerDied","Data":"4d2f981bd5dc73f066efa6d1e18c9fe94078161324783065a6f61a1b5339a3e9"}
Sep 17 03:40:36.047932 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:36.047824    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm" event={"ID":"aca02225-cef7-4f9c-8545-6c07bf293a3e","Type":"ContainerDied","Data":"5c350b3e16fc080787856a3d202e31b5e75252294e6bd1df9a651bf77357314b"}
Sep 17 03:40:36.047932 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:36.047844    2557 scope.go:117] "RemoveContainer" containerID="4d2f981bd5dc73f066efa6d1e18c9fe94078161324783065a6f61a1b5339a3e9"
Sep 17 03:40:36.061861 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:36.061836    2557 scope.go:117] "RemoveContainer" containerID="4d2f981bd5dc73f066efa6d1e18c9fe94078161324783065a6f61a1b5339a3e9"
Sep 17 03:40:36.062226 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:40:36.062182    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d2f981bd5dc73f066efa6d1e18c9fe94078161324783065a6f61a1b5339a3e9\": container with ID starting with 4d2f981bd5dc73f066efa6d1e18c9fe94078161324783065a6f61a1b5339a3e9 not found: ID does not exist" containerID="4d2f981bd5dc73f066efa6d1e18c9fe94078161324783065a6f61a1b5339a3e9"
Sep 17 03:40:36.062328 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:36.062237    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d2f981bd5dc73f066efa6d1e18c9fe94078161324783065a6f61a1b5339a3e9"} err="failed to get container status \"4d2f981bd5dc73f066efa6d1e18c9fe94078161324783065a6f61a1b5339a3e9\": rpc error: code = NotFound desc = could not find container \"4d2f981bd5dc73f066efa6d1e18c9fe94078161324783065a6f61a1b5339a3e9\": container with ID starting with 4d2f981bd5dc73f066efa6d1e18c9fe94078161324783065a6f61a1b5339a3e9 not found: ID does not exist"
Sep 17 03:40:36.069343 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:36.069322    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"]
Sep 17 03:40:36.072313 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:36.072295    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-214fea/e2e-conflict-a-fed91830-kserve-7b9d4f9685-hcvpm"]
Sep 17 03:40:37.916789 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:40:37.916756    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aca02225-cef7-4f9c-8545-6c07bf293a3e" path="/var/lib/kubelet/pods/aca02225-cef7-4f9c-8545-6c07bf293a3e/volumes"
Sep 17 03:41:08.943235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:08.943188    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"]
Sep 17 03:41:08.943802 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:08.943435    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh" podUID="55f3f719-8733-4c14-8d24-217810ed59a3" containerName="maas-api" containerID="cri-o://5b63c262305e24ec4fd073e61772cc9e48e88f0cb5d224c62885da6d1a8211f5" gracePeriod=30
Sep 17 03:41:09.022416 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.022343    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"]
Sep 17 03:41:09.022636 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.022607    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65" podUID="f9f91d55-34b9-4bc0-a668-c59c64f2428a" containerName="payload-pre-processing" containerID="cri-o://43455e4a32f4d12766b08070fbfe1bd77a7961bc7ac9b177b2aa88fb7e2e19e4" gracePeriod=30
Sep 17 03:41:09.240525 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.240489    2557 generic.go:356] "Generic (PLEG): container finished" podID="55f3f719-8733-4c14-8d24-217810ed59a3" containerID="5b63c262305e24ec4fd073e61772cc9e48e88f0cb5d224c62885da6d1a8211f5" exitCode=0
Sep 17 03:41:09.240691 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.240616    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh" event={"ID":"55f3f719-8733-4c14-8d24-217810ed59a3","Type":"ContainerDied","Data":"5b63c262305e24ec4fd073e61772cc9e48e88f0cb5d224c62885da6d1a8211f5"}
Sep 17 03:41:09.242568 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.242536    2557 generic.go:356] "Generic (PLEG): container finished" podID="f9f91d55-34b9-4bc0-a668-c59c64f2428a" containerID="43455e4a32f4d12766b08070fbfe1bd77a7961bc7ac9b177b2aa88fb7e2e19e4" exitCode=0
Sep 17 03:41:09.242674 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.242635    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65" event={"ID":"f9f91d55-34b9-4bc0-a668-c59c64f2428a","Type":"ContainerDied","Data":"43455e4a32f4d12766b08070fbfe1bd77a7961bc7ac9b177b2aa88fb7e2e19e4"}
Sep 17 03:41:09.308144 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.308119    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:41:09.311809 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.311788    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"
Sep 17 03:41:09.400505 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.400464    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkqr9\" (UniqueName: \"kubernetes.io/projected/55f3f719-8733-4c14-8d24-217810ed59a3-kube-api-access-zkqr9\") pod \"55f3f719-8733-4c14-8d24-217810ed59a3\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") "
Sep 17 03:41:09.400692 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.400516    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-metrics-certs\") pod \"55f3f719-8733-4c14-8d24-217810ed59a3\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") "
Sep 17 03:41:09.400692 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.400540    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-maas-api-tls\") pod \"55f3f719-8733-4c14-8d24-217810ed59a3\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") "
Sep 17 03:41:09.400692 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.400585    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f3f719-8733-4c14-8d24-217810ed59a3-openshift-service-ca\") pod \"55f3f719-8733-4c14-8d24-217810ed59a3\" (UID: \"55f3f719-8733-4c14-8d24-217810ed59a3\") "
Sep 17 03:41:09.401015 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.400984    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f3f719-8733-4c14-8d24-217810ed59a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "55f3f719-8733-4c14-8d24-217810ed59a3" (UID: "55f3f719-8733-4c14-8d24-217810ed59a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:41:09.402634 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.402602    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f3f719-8733-4c14-8d24-217810ed59a3-kube-api-access-zkqr9" (OuterVolumeSpecName: "kube-api-access-zkqr9") pod "55f3f719-8733-4c14-8d24-217810ed59a3" (UID: "55f3f719-8733-4c14-8d24-217810ed59a3"). InnerVolumeSpecName "kube-api-access-zkqr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:41:09.402737 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.402659    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "55f3f719-8733-4c14-8d24-217810ed59a3" (UID: "55f3f719-8733-4c14-8d24-217810ed59a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:41:09.402737 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.402685    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "55f3f719-8733-4c14-8d24-217810ed59a3" (UID: "55f3f719-8733-4c14-8d24-217810ed59a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:41:09.501633 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.501557    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2brpv\" (UniqueName: \"kubernetes.io/projected/f9f91d55-34b9-4bc0-a668-c59c64f2428a-kube-api-access-2brpv\") pod \"f9f91d55-34b9-4bc0-a668-c59c64f2428a\" (UID: \"f9f91d55-34b9-4bc0-a668-c59c64f2428a\") "
Sep 17 03:41:09.501633 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.501617    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f91d55-34b9-4bc0-a668-c59c64f2428a-plugins-config-volume\") pod \"f9f91d55-34b9-4bc0-a668-c59c64f2428a\" (UID: \"f9f91d55-34b9-4bc0-a668-c59c64f2428a\") "
Sep 17 03:41:09.502010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.501983    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkqr9\" (UniqueName: \"kubernetes.io/projected/55f3f719-8733-4c14-8d24-217810ed59a3-kube-api-access-zkqr9\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:41:09.502010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.502014    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-metrics-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:41:09.502250 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.502028    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55f3f719-8733-4c14-8d24-217810ed59a3-maas-api-tls\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:41:09.502250 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.502042    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55f3f719-8733-4c14-8d24-217810ed59a3-openshift-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:41:09.502250 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.502010    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f91d55-34b9-4bc0-a668-c59c64f2428a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9f91d55-34b9-4bc0-a668-c59c64f2428a" (UID: "f9f91d55-34b9-4bc0-a668-c59c64f2428a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:41:09.503516 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.503492    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f91d55-34b9-4bc0-a668-c59c64f2428a-kube-api-access-2brpv" (OuterVolumeSpecName: "kube-api-access-2brpv") pod "f9f91d55-34b9-4bc0-a668-c59c64f2428a" (UID: "f9f91d55-34b9-4bc0-a668-c59c64f2428a"). InnerVolumeSpecName "kube-api-access-2brpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:41:09.602749 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.602721    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f91d55-34b9-4bc0-a668-c59c64f2428a-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:41:09.602749 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:09.602747    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2brpv\" (UniqueName: \"kubernetes.io/projected/f9f91d55-34b9-4bc0-a668-c59c64f2428a-kube-api-access-2brpv\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:41:09.952577 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:41:09.952546    2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f91d55_34b9_4bc0_a668_c59c64f2428a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f3f719_8733_4c14_8d24_217810ed59a3.slice/crio-303ec81988825339afa538490fa4e12ac7520a55a5e4d6a3c97ad6c45761be69\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f3f719_8733_4c14_8d24_217810ed59a3.slice\": RecentStats: unable to find data in memory cache]"
Sep 17 03:41:10.148716 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:10.148687    2557 scope.go:117] "RemoveContainer" containerID="75a7846647f8d977b96d153160513a0ad0f123a1b1f456110928480cd29d290e"
Sep 17 03:41:10.249331 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:10.249273    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"
Sep 17 03:41:10.249331 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:10.249292    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65" event={"ID":"f9f91d55-34b9-4bc0-a668-c59c64f2428a","Type":"ContainerDied","Data":"f2c85e28d54c746f3630828dfdfbe3a4a55a5e00e88b309d772ca430762c6b5b"}
Sep 17 03:41:10.249518 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:10.249400    2557 scope.go:117] "RemoveContainer" containerID="43455e4a32f4d12766b08070fbfe1bd77a7961bc7ac9b177b2aa88fb7e2e19e4"
Sep 17 03:41:10.251299 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:10.251276    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh" event={"ID":"55f3f719-8733-4c14-8d24-217810ed59a3","Type":"ContainerDied","Data":"303ec81988825339afa538490fa4e12ac7520a55a5e4d6a3c97ad6c45761be69"}
Sep 17 03:41:10.251527 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:10.251354    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"
Sep 17 03:41:10.258052 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:10.258038    2557 scope.go:117] "RemoveContainer" containerID="5b63c262305e24ec4fd073e61772cc9e48e88f0cb5d224c62885da6d1a8211f5"
Sep 17 03:41:10.272543 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:10.272525    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"]
Sep 17 03:41:10.274822 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:10.274804    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c10ab8fb-5b7cf8d8db-mbm65"]
Sep 17 03:41:10.289136 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:10.289117    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"]
Sep 17 03:41:10.292547 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:10.292527    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c10ab8fb-78bc65f64f-t56mh"]
Sep 17 03:41:11.915723 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:11.915685    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55f3f719-8733-4c14-8d24-217810ed59a3" path="/var/lib/kubelet/pods/55f3f719-8733-4c14-8d24-217810ed59a3/volumes"
Sep 17 03:41:11.916268 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:41:11.916247    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9f91d55-34b9-4bc0-a668-c59c64f2428a" path="/var/lib/kubelet/pods/f9f91d55-34b9-4bc0-a668-c59c64f2428a/volumes"
Sep 17 03:43:09.233827 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.233738    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"]
Sep 17 03:43:09.234528 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.234508    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f3f719-8733-4c14-8d24-217810ed59a3" containerName="maas-api"
Sep 17 03:43:09.234624 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.234531    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f3f719-8733-4c14-8d24-217810ed59a3" containerName="maas-api"
Sep 17 03:43:09.234624 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.234551    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aca02225-cef7-4f9c-8545-6c07bf293a3e" containerName="main"
Sep 17 03:43:09.234624 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.234560    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="aca02225-cef7-4f9c-8545-6c07bf293a3e" containerName="main"
Sep 17 03:43:09.234624 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.234570    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9f91d55-34b9-4bc0-a668-c59c64f2428a" containerName="payload-pre-processing"
Sep 17 03:43:09.234624 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.234575    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9f91d55-34b9-4bc0-a668-c59c64f2428a" containerName="payload-pre-processing"
Sep 17 03:43:09.234886 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.234666    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9f91d55-34b9-4bc0-a668-c59c64f2428a" containerName="payload-pre-processing"
Sep 17 03:43:09.234886 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.234674    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="55f3f719-8733-4c14-8d24-217810ed59a3" containerName="maas-api"
Sep 17 03:43:09.234886 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.234680    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="aca02225-cef7-4f9c-8545-6c07bf293a3e" containerName="main"
Sep 17 03:43:09.238367 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.238345    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.240676 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.240658    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4a43bf-openshift-default-dockercfg-5c4nb\""
Sep 17 03:43:09.250661 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.250638    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"]
Sep 17 03:43:09.342043 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.342016    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-workload-socket\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.342171 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.342058    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-data\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.342171 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.342077    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-envoy\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.342343 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.342171    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-credential-socket\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.342343 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.342238    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-podinfo\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.342343 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.342292    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-token\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.342343 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.342323    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-workload-certs\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.342529 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.342346    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2xzt\" (UniqueName: \"kubernetes.io/projected/ed111905-0ad6-4068-8e56-90df7fd45dc4-kube-api-access-x2xzt\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.342529 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.342447    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed111905-0ad6-4068-8e56-90df7fd45dc4-istiod-ca-cert\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.445037 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.443542    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-workload-socket\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.445037 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.443611    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-data\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.445037 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.443644    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-envoy\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.445037 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.443702    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-credential-socket\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.445037 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.443726    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-podinfo\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.445037 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.443766    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-token\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.445037 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.443793    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-workload-certs\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.445037 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.443815    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2xzt\" (UniqueName: \"kubernetes.io/projected/ed111905-0ad6-4068-8e56-90df7fd45dc4-kube-api-access-x2xzt\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.445037 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.443911    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed111905-0ad6-4068-8e56-90df7fd45dc4-istiod-ca-cert\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.445037 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.443995    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-workload-socket\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.445037 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.444570    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-credential-socket\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.445037 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.444683    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed111905-0ad6-4068-8e56-90df7fd45dc4-istiod-ca-cert\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.445037 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.444783    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-workload-certs\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.445847 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.445164    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-data\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.447706 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.447678    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-envoy\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.448483 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.448426    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-podinfo\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.462050 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.462021    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2xzt\" (UniqueName: \"kubernetes.io/projected/ed111905-0ad6-4068-8e56-90df7fd45dc4-kube-api-access-x2xzt\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.462684 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.462663    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-token\") pod \"e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") " pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.549501 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.549443    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:09.831289 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.831255    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"]
Sep 17 03:43:09.832511 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:43:09.832487    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded111905_0ad6_4068_8e56_90df7fd45dc4.slice/crio-d6ab24d0b46eb4b13b5c50e59847494615fd8844eaf92ace7137a12f19b13baf WatchSource:0}: Error finding container d6ab24d0b46eb4b13b5c50e59847494615fd8844eaf92ace7137a12f19b13baf: Status 404 returned error can't find the container with id d6ab24d0b46eb4b13b5c50e59847494615fd8844eaf92ace7137a12f19b13baf
Sep 17 03:43:09.835321 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.835289    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:43:09.835436 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.835364    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:43:09.835436 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.835409    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:43:09.950918 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.950819    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t" event={"ID":"ed111905-0ad6-4068-8e56-90df7fd45dc4","Type":"ContainerStarted","Data":"71072c0d62247f01110f57d73cc030b83357b88c7fcac93a2dd201c2f613d43c"}
Sep 17 03:43:09.950918 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.950886    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t" event={"ID":"ed111905-0ad6-4068-8e56-90df7fd45dc4","Type":"ContainerStarted","Data":"d6ab24d0b46eb4b13b5c50e59847494615fd8844eaf92ace7137a12f19b13baf"}
Sep 17 03:43:09.975042 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:09.974983    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t" podStartSLOduration=0.97496514 podStartE2EDuration="974.96514ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:43:09.974156817 +0000 UTC m=+1440.577252822" watchObservedRunningTime="2026-09-17 03:43:09.97496514 +0000 UTC m=+1440.578061145"
Sep 17 03:43:10.549709 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:10.549678    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:10.556441 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:10.556416    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:10.957797 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:10.957764    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:10.958705 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:10.958680    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:16.774467 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:16.774432    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"]
Sep 17 03:43:16.779456 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:16.779430    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:16.781511 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:16.781489    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4a43bf-serving-cert\""
Sep 17 03:43:16.787770 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:16.787748    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"]
Sep 17 03:43:16.920987 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:16.920955    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae0183b3-188d-4f38-a159-18aa11c9bb29-metrics-certs\") pod \"maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:16.921178 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:16.921026    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0183b3-188d-4f38-a159-18aa11c9bb29-maas-api-tls\") pod \"maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:16.921178 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:16.921060    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-878f7\" (UniqueName: \"kubernetes.io/projected/ae0183b3-188d-4f38-a159-18aa11c9bb29-kube-api-access-878f7\") pod \"maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:16.921178 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:16.921084    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae0183b3-188d-4f38-a159-18aa11c9bb29-openshift-service-ca\") pod \"maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:17.021864 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.021828    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae0183b3-188d-4f38-a159-18aa11c9bb29-openshift-service-ca\") pod \"maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:17.022022 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.021921    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae0183b3-188d-4f38-a159-18aa11c9bb29-metrics-certs\") pod \"maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:17.022022 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.022015    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0183b3-188d-4f38-a159-18aa11c9bb29-maas-api-tls\") pod \"maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:17.022153 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.022083    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-878f7\" (UniqueName: \"kubernetes.io/projected/ae0183b3-188d-4f38-a159-18aa11c9bb29-kube-api-access-878f7\") pod \"maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:17.022653 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.022631    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae0183b3-188d-4f38-a159-18aa11c9bb29-openshift-service-ca\") pod \"maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:17.024440 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.024419    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae0183b3-188d-4f38-a159-18aa11c9bb29-metrics-certs\") pod \"maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:17.024558 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.024469    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0183b3-188d-4f38-a159-18aa11c9bb29-maas-api-tls\") pod \"maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:17.029444 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.029421    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-878f7\" (UniqueName: \"kubernetes.io/projected/ae0183b3-188d-4f38-a159-18aa11c9bb29-kube-api-access-878f7\") pod \"maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:17.094734 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.094709    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:17.372841 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.372768    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"]
Sep 17 03:43:17.375536 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:43:17.375509    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0183b3_188d_4f38_a159_18aa11c9bb29.slice/crio-7c3ca3576d1722246862c3ab0e44c2a66efc1e16730c012801c2123b2092cf98 WatchSource:0}: Error finding container 7c3ca3576d1722246862c3ab0e44c2a66efc1e16730c012801c2123b2092cf98: Status 404 returned error can't find the container with id 7c3ca3576d1722246862c3ab0e44c2a66efc1e16730c012801c2123b2092cf98
Sep 17 03:43:17.377255 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.377238    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 03:43:17.399425 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.399386    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"]
Sep 17 03:43:17.404137 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.404121    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"
Sep 17 03:43:17.406824 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.406807    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4a43bf-dockercfg-7crvr\""
Sep 17 03:43:17.407098 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.407078    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4a43bf\""
Sep 17 03:43:17.415250 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.415231    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"]
Sep 17 03:43:17.527255 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.527219    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl52f\" (UniqueName: \"kubernetes.io/projected/b1cb8456-384c-43d5-b904-d840ae827e59-kube-api-access-sl52f\") pod \"payload-processing-e2e-api-a-4a43bf-95c976b95-284rp\" (UID: \"b1cb8456-384c-43d5-b904-d840ae827e59\") " pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"
Sep 17 03:43:17.527478 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.527457    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cb8456-384c-43d5-b904-d840ae827e59-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4a43bf-95c976b95-284rp\" (UID: \"b1cb8456-384c-43d5-b904-d840ae827e59\") " pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"
Sep 17 03:43:17.628687 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.628597    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl52f\" (UniqueName: \"kubernetes.io/projected/b1cb8456-384c-43d5-b904-d840ae827e59-kube-api-access-sl52f\") pod \"payload-processing-e2e-api-a-4a43bf-95c976b95-284rp\" (UID: \"b1cb8456-384c-43d5-b904-d840ae827e59\") " pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"
Sep 17 03:43:17.628881 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.628800    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cb8456-384c-43d5-b904-d840ae827e59-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4a43bf-95c976b95-284rp\" (UID: \"b1cb8456-384c-43d5-b904-d840ae827e59\") " pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"
Sep 17 03:43:17.629494 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.629471    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cb8456-384c-43d5-b904-d840ae827e59-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4a43bf-95c976b95-284rp\" (UID: \"b1cb8456-384c-43d5-b904-d840ae827e59\") " pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"
Sep 17 03:43:17.636655 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.636631    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl52f\" (UniqueName: \"kubernetes.io/projected/b1cb8456-384c-43d5-b904-d840ae827e59-kube-api-access-sl52f\") pod \"payload-processing-e2e-api-a-4a43bf-95c976b95-284rp\" (UID: \"b1cb8456-384c-43d5-b904-d840ae827e59\") " pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"
Sep 17 03:43:17.717405 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.717378    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"
Sep 17 03:43:17.988517 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.988493    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"]
Sep 17 03:43:17.990128 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:43:17.990108    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb8456_384c_43d5_b904_d840ae827e59.slice/crio-ffead8d9983960789f04b0309beb7b8744eb1f720c5564d4843925dc91f852bf WatchSource:0}: Error finding container ffead8d9983960789f04b0309beb7b8744eb1f720c5564d4843925dc91f852bf: Status 404 returned error can't find the container with id ffead8d9983960789f04b0309beb7b8744eb1f720c5564d4843925dc91f852bf
Sep 17 03:43:17.999672 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.999641    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4" event={"ID":"ae0183b3-188d-4f38-a159-18aa11c9bb29","Type":"ContainerStarted","Data":"3d5de7a61bf19ef532155351087d70a5ff5793a1ef9e99a646ee3a741cb06e07"}
Sep 17 03:43:17.999784 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.999696    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4" event={"ID":"ae0183b3-188d-4f38-a159-18aa11c9bb29","Type":"ContainerStarted","Data":"7c3ca3576d1722246862c3ab0e44c2a66efc1e16730c012801c2123b2092cf98"}
Sep 17 03:43:17.999784 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:17.999725    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:18.001038 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:18.001012    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp" event={"ID":"b1cb8456-384c-43d5-b904-d840ae827e59","Type":"ContainerStarted","Data":"ffead8d9983960789f04b0309beb7b8744eb1f720c5564d4843925dc91f852bf"}
Sep 17 03:43:18.016336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:18.016295    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4" podStartSLOduration=1.547180322 podStartE2EDuration="2.016280228s" totalImagesPullingTime="469.099906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:43:18.014637315 +0000 UTC m=+1448.617733321" watchObservedRunningTime="2026-09-17 03:43:18.016280228 +0000 UTC m=+1448.619376231"
Sep 17 03:43:19.009421 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:19.009384    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp" event={"ID":"b1cb8456-384c-43d5-b904-d840ae827e59","Type":"ContainerStarted","Data":"477c4d204afbd68035ce37cdedd4b008d7c57e1b4206512c238b824412d8467c"}
Sep 17 03:43:19.009799 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:19.009463    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"
Sep 17 03:43:19.026847 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:19.026789    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp" podStartSLOduration=2.026773719 podStartE2EDuration="2.026773719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:43:19.025062907 +0000 UTC m=+1449.628158908" watchObservedRunningTime="2026-09-17 03:43:19.026773719 +0000 UTC m=+1449.629869722"
Sep 17 03:43:24.014907 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:24.014874    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:43:30.017631 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:30.017600    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"
Sep 17 03:43:50.215300 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:50.215264    2557 generic.go:356] "Generic (PLEG): container finished" podID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerID="71072c0d62247f01110f57d73cc030b83357b88c7fcac93a2dd201c2f613d43c" exitCode=0
Sep 17 03:43:50.215754 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:50.215306    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t" event={"ID":"ed111905-0ad6-4068-8e56-90df7fd45dc4","Type":"ContainerDied","Data":"71072c0d62247f01110f57d73cc030b83357b88c7fcac93a2dd201c2f613d43c"}
Sep 17 03:43:50.215754 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:50.215655    2557 scope.go:117] "RemoveContainer" containerID="71072c0d62247f01110f57d73cc030b83357b88c7fcac93a2dd201c2f613d43c"
Sep 17 03:43:50.216743 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:50.216711    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:43:50.216797 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:50.216779    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:43:50.216836 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:50.216826    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:43:51.223877 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:51.223830    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t" event={"ID":"ed111905-0ad6-4068-8e56-90df7fd45dc4","Type":"ContainerStarted","Data":"89c1400cac61dd09ee7867a84f2db42379e1fe45b1ae5a28658cb04c3abcead1"}
Sep 17 03:43:51.549834 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:51.549758    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:51.550654 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:51.550632    2557 patch_prober.go:28] interesting pod/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 17 03:43:51.550727 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:51.550687    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t" podUID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 17 03:43:52.550755 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:52.550708    2557 patch_prober.go:28] interesting pod/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 17 03:43:52.551327 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:52.550800    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t" podUID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 17 03:43:53.550689 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:53.550655    2557 patch_prober.go:28] interesting pod/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 17 03:43:53.550928 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:53.550713    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t" podUID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 17 03:43:54.550680 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:54.550643    2557 patch_prober.go:28] interesting pod/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 17 03:43:54.550855 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:54.550699    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t" podUID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 17 03:43:55.551134 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:55.550719    2557 patch_prober.go:28] interesting pod/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 17 03:43:55.551134 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:55.550784    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t" podUID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 17 03:43:55.957961 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:55.957927    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:56.550181 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:56.550147    2557 patch_prober.go:28] interesting pod/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Sep 17 03:43:56.550420 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:56.550223    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t" podUID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Sep 17 03:43:57.553615 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:57.553588    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:43:57.554480 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:43:57.554460    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:44:02.042003 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:02.041964    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j"]
Sep 17 03:44:02.046884 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:02.046865    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j"
Sep 17 03:44:02.049220 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:02.049176    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:44:02.052886 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:02.052859    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j"]
Sep 17 03:44:02.167318 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:02.167291    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0c8786-f60e-425a-a9e7-4d9557ad8836-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j\" (UID: \"0c0c8786-f60e-425a-a9e7-4d9557ad8836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j"
Sep 17 03:44:02.268587 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:02.268561    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0c8786-f60e-425a-a9e7-4d9557ad8836-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j\" (UID: \"0c0c8786-f60e-425a-a9e7-4d9557ad8836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j"
Sep 17 03:44:02.269154 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:02.269132    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0c8786-f60e-425a-a9e7-4d9557ad8836-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j\" (UID: \"0c0c8786-f60e-425a-a9e7-4d9557ad8836\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j"
Sep 17 03:44:02.360950 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:02.360886    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j"
Sep 17 03:44:02.638129 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:02.638101    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j"]
Sep 17 03:44:02.639084 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:44:02.639058    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0c8786_f60e_425a_a9e7_4d9557ad8836.slice/crio-82aca1f03a5fec205447683cabaa8f697a5bb1526fa7592244e1a195c4ee22d2 WatchSource:0}: Error finding container 82aca1f03a5fec205447683cabaa8f697a5bb1526fa7592244e1a195c4ee22d2: Status 404 returned error can't find the container with id 82aca1f03a5fec205447683cabaa8f697a5bb1526fa7592244e1a195c4ee22d2
Sep 17 03:44:03.319744 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:03.319716    2557 generic.go:356] "Generic (PLEG): container finished" podID="0c0c8786-f60e-425a-a9e7-4d9557ad8836" containerID="dd1bd18e9a13a62a73581855e6cd2c5bab4b10a67934a308b525d8487684b255" exitCode=0
Sep 17 03:44:03.320029 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:03.319804    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j" event={"ID":"0c0c8786-f60e-425a-a9e7-4d9557ad8836","Type":"ContainerDied","Data":"dd1bd18e9a13a62a73581855e6cd2c5bab4b10a67934a308b525d8487684b255"}
Sep 17 03:44:03.320029 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:03.319862    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j" event={"ID":"0c0c8786-f60e-425a-a9e7-4d9557ad8836","Type":"ContainerStarted","Data":"82aca1f03a5fec205447683cabaa8f697a5bb1526fa7592244e1a195c4ee22d2"}
Sep 17 03:44:04.471763 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:04.471735    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j"
Sep 17 03:44:04.597102 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:04.597042    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0c8786-f60e-425a-a9e7-4d9557ad8836-maas-api-service-ca\") pod \"0c0c8786-f60e-425a-a9e7-4d9557ad8836\" (UID: \"0c0c8786-f60e-425a-a9e7-4d9557ad8836\") "
Sep 17 03:44:04.597395 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:04.597376    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0c8786-f60e-425a-a9e7-4d9557ad8836-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0c0c8786-f60e-425a-a9e7-4d9557ad8836" (UID: "0c0c8786-f60e-425a-a9e7-4d9557ad8836"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:44:04.698776 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:04.698743    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c0c8786-f60e-425a-a9e7-4d9557ad8836-maas-api-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:05.333942 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:05.333916    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j"
Sep 17 03:44:05.334102 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:05.333939    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j" event={"ID":"0c0c8786-f60e-425a-a9e7-4d9557ad8836","Type":"ContainerDied","Data":"82aca1f03a5fec205447683cabaa8f697a5bb1526fa7592244e1a195c4ee22d2"}
Sep 17 03:44:05.334102 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:05.333994    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82aca1f03a5fec205447683cabaa8f697a5bb1526fa7592244e1a195c4ee22d2"
Sep 17 03:44:06.976536 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:06.976507    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"]
Sep 17 03:44:09.980469 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:09.980441    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:44:09.985929 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:09.985907    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:44:13.555122 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:13.555087    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m"]
Sep 17 03:44:13.555593 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:13.555580    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c0c8786-f60e-425a-a9e7-4d9557ad8836" containerName="revoke-keys"
Sep 17 03:44:13.555651 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:13.555594    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c0c8786-f60e-425a-a9e7-4d9557ad8836" containerName="revoke-keys"
Sep 17 03:44:13.555688 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:13.555677    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c0c8786-f60e-425a-a9e7-4d9557ad8836" containerName="revoke-keys"
Sep 17 03:44:13.558797 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:13.558782    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m"
Sep 17 03:44:13.560994 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:13.560972    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:44:13.565489 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:13.565469    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m"]
Sep 17 03:44:13.691428 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:13.691393    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55405e79-549b-49d5-9b5a-b9c8e3710cdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m\" (UID: \"55405e79-549b-49d5-9b5a-b9c8e3710cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m"
Sep 17 03:44:13.792656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:13.792630    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55405e79-549b-49d5-9b5a-b9c8e3710cdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m\" (UID: \"55405e79-549b-49d5-9b5a-b9c8e3710cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m"
Sep 17 03:44:13.793242 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:13.793223    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55405e79-549b-49d5-9b5a-b9c8e3710cdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m\" (UID: \"55405e79-549b-49d5-9b5a-b9c8e3710cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m"
Sep 17 03:44:13.872279 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:13.872257    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m"
Sep 17 03:44:14.153813 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:14.153787    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m"]
Sep 17 03:44:14.154393 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:44:14.154370    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55405e79_549b_49d5_9b5a_b9c8e3710cdb.slice/crio-fc51d69dc54a0b6b2b701861d8c556fad79816dbe89895e537754ff554f2fd49 WatchSource:0}: Error finding container fc51d69dc54a0b6b2b701861d8c556fad79816dbe89895e537754ff554f2fd49: Status 404 returned error can't find the container with id fc51d69dc54a0b6b2b701861d8c556fad79816dbe89895e537754ff554f2fd49
Sep 17 03:44:14.388266 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:14.388239    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m" event={"ID":"55405e79-549b-49d5-9b5a-b9c8e3710cdb","Type":"ContainerStarted","Data":"39d017df5a9f935bf7c4968a2437f5f3d2c9da0a91270b5ed7976219ab75dcb3"}
Sep 17 03:44:14.388377 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:14.388293    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m" event={"ID":"55405e79-549b-49d5-9b5a-b9c8e3710cdb","Type":"ContainerStarted","Data":"fc51d69dc54a0b6b2b701861d8c556fad79816dbe89895e537754ff554f2fd49"}
Sep 17 03:44:14.404171 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:14.404087    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m" podStartSLOduration=1.404074329 podStartE2EDuration="1.404074329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:44:14.401556629 +0000 UTC m=+1505.004652633" watchObservedRunningTime="2026-09-17 03:44:14.404074329 +0000 UTC m=+1505.007170332"
Sep 17 03:44:15.395019 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:15.394984    2557 generic.go:356] "Generic (PLEG): container finished" podID="55405e79-549b-49d5-9b5a-b9c8e3710cdb" containerID="39d017df5a9f935bf7c4968a2437f5f3d2c9da0a91270b5ed7976219ab75dcb3" exitCode=0
Sep 17 03:44:15.395411 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:15.395064    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m" event={"ID":"55405e79-549b-49d5-9b5a-b9c8e3710cdb","Type":"ContainerDied","Data":"39d017df5a9f935bf7c4968a2437f5f3d2c9da0a91270b5ed7976219ab75dcb3"}
Sep 17 03:44:16.544858 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:16.544835    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m"
Sep 17 03:44:16.730591 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:16.730518    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55405e79-549b-49d5-9b5a-b9c8e3710cdb-maas-api-service-ca\") pod \"55405e79-549b-49d5-9b5a-b9c8e3710cdb\" (UID: \"55405e79-549b-49d5-9b5a-b9c8e3710cdb\") "
Sep 17 03:44:16.730918 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:16.730890    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55405e79-549b-49d5-9b5a-b9c8e3710cdb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55405e79-549b-49d5-9b5a-b9c8e3710cdb" (UID: "55405e79-549b-49d5-9b5a-b9c8e3710cdb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:44:16.831465 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:16.831440    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55405e79-549b-49d5-9b5a-b9c8e3710cdb-maas-api-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:17.408683 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:17.408657    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m"
Sep 17 03:44:17.408983 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:17.408657    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m" event={"ID":"55405e79-549b-49d5-9b5a-b9c8e3710cdb","Type":"ContainerDied","Data":"fc51d69dc54a0b6b2b701861d8c556fad79816dbe89895e537754ff554f2fd49"}
Sep 17 03:44:17.408983 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:17.408782    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc51d69dc54a0b6b2b701861d8c556fad79816dbe89895e537754ff554f2fd49"
Sep 17 03:44:21.078483 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:21.078449    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j"]
Sep 17 03:44:21.082470 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:21.082442    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1afd34-gvw4j"]
Sep 17 03:44:21.916988 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:21.916951    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c0c8786-f60e-425a-a9e7-4d9557ad8836" path="/var/lib/kubelet/pods/0c0c8786-f60e-425a-a9e7-4d9557ad8836/volumes"
Sep 17 03:44:21.996117 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:21.996080    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt" podUID="c4a8a698-79e4-48ad-984b-a1709305ff8c" containerName="main" containerID="cri-o://16fb21e53a020a484644993c2860ac91511b82f1ce804d9c1b7d51b9c8b7341e" gracePeriod=45
Sep 17 03:44:22.267898 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.267876    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:44:22.286737 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.286707    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97fbb\" (UniqueName: \"kubernetes.io/projected/c4a8a698-79e4-48ad-984b-a1709305ff8c-kube-api-access-97fbb\") pod \"c4a8a698-79e4-48ad-984b-a1709305ff8c\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") "
Sep 17 03:44:22.286961 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.286754    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a8a698-79e4-48ad-984b-a1709305ff8c-tls-certs\") pod \"c4a8a698-79e4-48ad-984b-a1709305ff8c\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") "
Sep 17 03:44:22.286961 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.286799    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-dshm\") pod \"c4a8a698-79e4-48ad-984b-a1709305ff8c\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") "
Sep 17 03:44:22.286961 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.286827    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-home\") pod \"c4a8a698-79e4-48ad-984b-a1709305ff8c\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") "
Sep 17 03:44:22.286961 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.286856    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-model-cache\") pod \"c4a8a698-79e4-48ad-984b-a1709305ff8c\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") "
Sep 17 03:44:22.286961 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.286927    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-tmp-dir\") pod \"c4a8a698-79e4-48ad-984b-a1709305ff8c\" (UID: \"c4a8a698-79e4-48ad-984b-a1709305ff8c\") "
Sep 17 03:44:22.287659 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.287382    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-home" (OuterVolumeSpecName: "home") pod "c4a8a698-79e4-48ad-984b-a1709305ff8c" (UID: "c4a8a698-79e4-48ad-984b-a1709305ff8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.287659 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.287393    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c4a8a698-79e4-48ad-984b-a1709305ff8c" (UID: "c4a8a698-79e4-48ad-984b-a1709305ff8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.287659 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.287633    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c4a8a698-79e4-48ad-984b-a1709305ff8c" (UID: "c4a8a698-79e4-48ad-984b-a1709305ff8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.289727 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.289690    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a8a698-79e4-48ad-984b-a1709305ff8c-kube-api-access-97fbb" (OuterVolumeSpecName: "kube-api-access-97fbb") pod "c4a8a698-79e4-48ad-984b-a1709305ff8c" (UID: "c4a8a698-79e4-48ad-984b-a1709305ff8c"). InnerVolumeSpecName "kube-api-access-97fbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:44:22.290178 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.290153    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4a8a698-79e4-48ad-984b-a1709305ff8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c4a8a698-79e4-48ad-984b-a1709305ff8c" (UID: "c4a8a698-79e4-48ad-984b-a1709305ff8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:44:22.290744 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.290708    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-dshm" (OuterVolumeSpecName: "dshm") pod "c4a8a698-79e4-48ad-984b-a1709305ff8c" (UID: "c4a8a698-79e4-48ad-984b-a1709305ff8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:22.388309 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.388284    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97fbb\" (UniqueName: \"kubernetes.io/projected/c4a8a698-79e4-48ad-984b-a1709305ff8c-kube-api-access-97fbb\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.388309 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.388310    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a8a698-79e4-48ad-984b-a1709305ff8c-tls-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.388472 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.388319    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-dshm\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.388472 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.388327    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-home\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.388472 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.388335    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-model-cache\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.388472 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.388343    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a8a698-79e4-48ad-984b-a1709305ff8c-tmp-dir\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:22.439987 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.439907    2557 generic.go:356] "Generic (PLEG): container finished" podID="c4a8a698-79e4-48ad-984b-a1709305ff8c" containerID="16fb21e53a020a484644993c2860ac91511b82f1ce804d9c1b7d51b9c8b7341e" exitCode=0
Sep 17 03:44:22.439987 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.439953    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"
Sep 17 03:44:22.440216 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.439990    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt" event={"ID":"c4a8a698-79e4-48ad-984b-a1709305ff8c","Type":"ContainerDied","Data":"16fb21e53a020a484644993c2860ac91511b82f1ce804d9c1b7d51b9c8b7341e"}
Sep 17 03:44:22.440216 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.440033    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt" event={"ID":"c4a8a698-79e4-48ad-984b-a1709305ff8c","Type":"ContainerDied","Data":"344fbb63340e848a53ec9d3213d8d0e98e009616759f31c73ecb53b3bcf25088"}
Sep 17 03:44:22.440216 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.440053    2557 scope.go:117] "RemoveContainer" containerID="16fb21e53a020a484644993c2860ac91511b82f1ce804d9c1b7d51b9c8b7341e"
Sep 17 03:44:22.451135 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.451121    2557 scope.go:117] "RemoveContainer" containerID="16fb21e53a020a484644993c2860ac91511b82f1ce804d9c1b7d51b9c8b7341e"
Sep 17 03:44:22.451445 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:44:22.451418    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fb21e53a020a484644993c2860ac91511b82f1ce804d9c1b7d51b9c8b7341e\": container with ID starting with 16fb21e53a020a484644993c2860ac91511b82f1ce804d9c1b7d51b9c8b7341e not found: ID does not exist" containerID="16fb21e53a020a484644993c2860ac91511b82f1ce804d9c1b7d51b9c8b7341e"
Sep 17 03:44:22.451533 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.451455    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fb21e53a020a484644993c2860ac91511b82f1ce804d9c1b7d51b9c8b7341e"} err="failed to get container status \"16fb21e53a020a484644993c2860ac91511b82f1ce804d9c1b7d51b9c8b7341e\": rpc error: code = NotFound desc = could not find container \"16fb21e53a020a484644993c2860ac91511b82f1ce804d9c1b7d51b9c8b7341e\": container with ID starting with 16fb21e53a020a484644993c2860ac91511b82f1ce804d9c1b7d51b9c8b7341e not found: ID does not exist"
Sep 17 03:44:22.462314 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.462293    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"]
Sep 17 03:44:22.466010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:22.465988    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1afd34/distinct-w0-1afd34-kserve-6d7b57d648-t29zt"]
Sep 17 03:44:23.916987 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:23.916955    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4a8a698-79e4-48ad-984b-a1709305ff8c" path="/var/lib/kubelet/pods/c4a8a698-79e4-48ad-984b-a1709305ff8c/volumes"
Sep 17 03:44:27.717635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:27.717608    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"]
Sep 17 03:44:27.718030 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:27.717798    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf" podUID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerName="istio-proxy" containerID="cri-o://bd85ddf58d29bde173534d84e9336befc59d9c1fb505e6fd8e1c7ba9da0aca10" gracePeriod=30
Sep 17 03:44:29.087270 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:29.087239    2557 patch_prober.go:28] interesting pod/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 03:44:29.087623 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:29.087302    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf" podUID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 03:44:32.167533 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.167504    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"]
Sep 17 03:44:32.168010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.167768    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4" podUID="ae0183b3-188d-4f38-a159-18aa11c9bb29" containerName="maas-api" containerID="cri-o://3d5de7a61bf19ef532155351087d70a5ff5793a1ef9e99a646ee3a741cb06e07" gracePeriod=30
Sep 17 03:44:32.225364 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.225323    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"]
Sep 17 03:44:32.225778 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.225748    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp" podUID="b1cb8456-384c-43d5-b904-d840ae827e59" containerName="payload-processing" containerID="cri-o://477c4d204afbd68035ce37cdedd4b008d7c57e1b4206512c238b824412d8467c" gracePeriod=30
Sep 17 03:44:32.385651 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.385607    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m"]
Sep 17 03:44:32.390187 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.390164    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4a43bf-9zv5m"]
Sep 17 03:44:32.522590 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.518874    2557 generic.go:356] "Generic (PLEG): container finished" podID="ae0183b3-188d-4f38-a159-18aa11c9bb29" containerID="3d5de7a61bf19ef532155351087d70a5ff5793a1ef9e99a646ee3a741cb06e07" exitCode=0
Sep 17 03:44:32.522590 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.518976    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4" event={"ID":"ae0183b3-188d-4f38-a159-18aa11c9bb29","Type":"ContainerDied","Data":"3d5de7a61bf19ef532155351087d70a5ff5793a1ef9e99a646ee3a741cb06e07"}
Sep 17 03:44:32.522590 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.519028    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4" event={"ID":"ae0183b3-188d-4f38-a159-18aa11c9bb29","Type":"ContainerDied","Data":"7c3ca3576d1722246862c3ab0e44c2a66efc1e16730c012801c2123b2092cf98"}
Sep 17 03:44:32.522590 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.519054    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c3ca3576d1722246862c3ab0e44c2a66efc1e16730c012801c2123b2092cf98"
Sep 17 03:44:32.522590 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.522157    2557 generic.go:356] "Generic (PLEG): container finished" podID="b1cb8456-384c-43d5-b904-d840ae827e59" containerID="477c4d204afbd68035ce37cdedd4b008d7c57e1b4206512c238b824412d8467c" exitCode=0
Sep 17 03:44:32.522590 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.522357    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp" event={"ID":"b1cb8456-384c-43d5-b904-d840ae827e59","Type":"ContainerDied","Data":"477c4d204afbd68035ce37cdedd4b008d7c57e1b4206512c238b824412d8467c"}
Sep 17 03:44:32.535795 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.535765    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:44:32.564432 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.564398    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"
Sep 17 03:44:32.597088 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.596399    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-878f7\" (UniqueName: \"kubernetes.io/projected/ae0183b3-188d-4f38-a159-18aa11c9bb29-kube-api-access-878f7\") pod \"ae0183b3-188d-4f38-a159-18aa11c9bb29\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") "
Sep 17 03:44:32.597088 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.596604    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae0183b3-188d-4f38-a159-18aa11c9bb29-openshift-service-ca\") pod \"ae0183b3-188d-4f38-a159-18aa11c9bb29\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") "
Sep 17 03:44:32.597088 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.596636    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0183b3-188d-4f38-a159-18aa11c9bb29-maas-api-tls\") pod \"ae0183b3-188d-4f38-a159-18aa11c9bb29\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") "
Sep 17 03:44:32.597088 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.596683    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae0183b3-188d-4f38-a159-18aa11c9bb29-metrics-certs\") pod \"ae0183b3-188d-4f38-a159-18aa11c9bb29\" (UID: \"ae0183b3-188d-4f38-a159-18aa11c9bb29\") "
Sep 17 03:44:32.597654 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.597614    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae0183b3-188d-4f38-a159-18aa11c9bb29-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ae0183b3-188d-4f38-a159-18aa11c9bb29" (UID: "ae0183b3-188d-4f38-a159-18aa11c9bb29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:44:32.599969 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.599941    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae0183b3-188d-4f38-a159-18aa11c9bb29-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ae0183b3-188d-4f38-a159-18aa11c9bb29" (UID: "ae0183b3-188d-4f38-a159-18aa11c9bb29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:44:32.600707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.600661    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae0183b3-188d-4f38-a159-18aa11c9bb29-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ae0183b3-188d-4f38-a159-18aa11c9bb29" (UID: "ae0183b3-188d-4f38-a159-18aa11c9bb29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:44:32.600817 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.600737    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0183b3-188d-4f38-a159-18aa11c9bb29-kube-api-access-878f7" (OuterVolumeSpecName: "kube-api-access-878f7") pod "ae0183b3-188d-4f38-a159-18aa11c9bb29" (UID: "ae0183b3-188d-4f38-a159-18aa11c9bb29"). InnerVolumeSpecName "kube-api-access-878f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:44:32.697760 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.697616    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cb8456-384c-43d5-b904-d840ae827e59-plugins-config-volume\") pod \"b1cb8456-384c-43d5-b904-d840ae827e59\" (UID: \"b1cb8456-384c-43d5-b904-d840ae827e59\") "
Sep 17 03:44:32.697932 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.697758    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl52f\" (UniqueName: \"kubernetes.io/projected/b1cb8456-384c-43d5-b904-d840ae827e59-kube-api-access-sl52f\") pod \"b1cb8456-384c-43d5-b904-d840ae827e59\" (UID: \"b1cb8456-384c-43d5-b904-d840ae827e59\") "
Sep 17 03:44:32.698083 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.698050    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1cb8456-384c-43d5-b904-d840ae827e59-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b1cb8456-384c-43d5-b904-d840ae827e59" (UID: "b1cb8456-384c-43d5-b904-d840ae827e59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:44:32.698262 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.698242    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1cb8456-384c-43d5-b904-d840ae827e59-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:32.698324 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.698267    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-878f7\" (UniqueName: \"kubernetes.io/projected/ae0183b3-188d-4f38-a159-18aa11c9bb29-kube-api-access-878f7\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:32.698324 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.698277    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae0183b3-188d-4f38-a159-18aa11c9bb29-openshift-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:32.698324 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.698286    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0183b3-188d-4f38-a159-18aa11c9bb29-maas-api-tls\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:32.698324 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.698300    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae0183b3-188d-4f38-a159-18aa11c9bb29-metrics-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:32.700208 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.700168    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cb8456-384c-43d5-b904-d840ae827e59-kube-api-access-sl52f" (OuterVolumeSpecName: "kube-api-access-sl52f") pod "b1cb8456-384c-43d5-b904-d840ae827e59" (UID: "b1cb8456-384c-43d5-b904-d840ae827e59"). InnerVolumeSpecName "kube-api-access-sl52f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:44:32.799371 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.799346    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl52f\" (UniqueName: \"kubernetes.io/projected/b1cb8456-384c-43d5-b904-d840ae827e59-kube-api-access-sl52f\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:32.979701 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:32.979674    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:44:33.102767 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.102737    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istiod-ca-cert\") pod \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") "
Sep 17 03:44:33.102767 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.102770    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-envoy\") pod \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") "
Sep 17 03:44:33.103014 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.102796    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-podinfo\") pod \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") "
Sep 17 03:44:33.103014 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.102822    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-data\") pod \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") "
Sep 17 03:44:33.103014 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.102849    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-workload-socket\") pod \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") "
Sep 17 03:44:33.103014 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.102872    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm9z9\" (UniqueName: \"kubernetes.io/projected/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-kube-api-access-dm9z9\") pod \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") "
Sep 17 03:44:33.103014 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.102955    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-workload-certs\") pod \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") "
Sep 17 03:44:33.103014 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.103009    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-credential-socket\") pod \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") "
Sep 17 03:44:33.103378 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.103034    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-token\") pod \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\" (UID: \"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94\") "
Sep 17 03:44:33.103378 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.103155    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-data" (OuterVolumeSpecName: "istio-data") pod "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" (UID: "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:33.103378 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.103224    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" (UID: "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:33.103378 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.103282    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" (UID: "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:44:33.103596 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.103425    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" (UID: "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:33.103596 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.103474    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" (UID: "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:33.103712 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.103610    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:33.103712 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.103629    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:33.103712 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.103644    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:33.103712 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.103657    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:33.103712 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.103670    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:33.105151 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.105048    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" (UID: "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:44:33.105151 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.105119    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" (UID: "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:33.105332 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.105149    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-kube-api-access-dm9z9" (OuterVolumeSpecName: "kube-api-access-dm9z9") pod "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" (UID: "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94"). InnerVolumeSpecName "kube-api-access-dm9z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:44:33.105332 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.105174    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-token" (OuterVolumeSpecName: "istio-token") pod "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" (UID: "4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:44:33.204497 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.204474    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:33.204497 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.204495    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:33.204800 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.204503    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:33.204800 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.204512    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm9z9\" (UniqueName: \"kubernetes.io/projected/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94-kube-api-access-dm9z9\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:33.532564 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.532526    2557 generic.go:356] "Generic (PLEG): container finished" podID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerID="bd85ddf58d29bde173534d84e9336befc59d9c1fb505e6fd8e1c7ba9da0aca10" exitCode=0
Sep 17 03:44:33.532750 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.532595    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"
Sep 17 03:44:33.532750 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.532639    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf" event={"ID":"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94","Type":"ContainerDied","Data":"bd85ddf58d29bde173534d84e9336befc59d9c1fb505e6fd8e1c7ba9da0aca10"}
Sep 17 03:44:33.532750 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.532730    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf" event={"ID":"4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94","Type":"ContainerDied","Data":"c334f564e4ecf12660a14d9db9be33e996cb117329c1f80622eed65238bead60"}
Sep 17 03:44:33.532959 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.532787    2557 scope.go:117] "RemoveContainer" containerID="bd85ddf58d29bde173534d84e9336befc59d9c1fb505e6fd8e1c7ba9da0aca10"
Sep 17 03:44:33.536004 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.534912    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"
Sep 17 03:44:33.536004 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.534987    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp" event={"ID":"b1cb8456-384c-43d5-b904-d840ae827e59","Type":"ContainerDied","Data":"ffead8d9983960789f04b0309beb7b8744eb1f720c5564d4843925dc91f852bf"}
Sep 17 03:44:33.536004 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.535635    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"
Sep 17 03:44:33.548142 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.548117    2557 scope.go:117] "RemoveContainer" containerID="d3cc3ab52306b191727104f4d1392040c6fd411795a4a86250da2f3af299f728"
Sep 17 03:44:33.560674 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.560220    2557 scope.go:117] "RemoveContainer" containerID="bd85ddf58d29bde173534d84e9336befc59d9c1fb505e6fd8e1c7ba9da0aca10"
Sep 17 03:44:33.560991 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:44:33.560967    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd85ddf58d29bde173534d84e9336befc59d9c1fb505e6fd8e1c7ba9da0aca10\": container with ID starting with bd85ddf58d29bde173534d84e9336befc59d9c1fb505e6fd8e1c7ba9da0aca10 not found: ID does not exist" containerID="bd85ddf58d29bde173534d84e9336befc59d9c1fb505e6fd8e1c7ba9da0aca10"
Sep 17 03:44:33.561088 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.561002    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd85ddf58d29bde173534d84e9336befc59d9c1fb505e6fd8e1c7ba9da0aca10"} err="failed to get container status \"bd85ddf58d29bde173534d84e9336befc59d9c1fb505e6fd8e1c7ba9da0aca10\": rpc error: code = NotFound desc = could not find container \"bd85ddf58d29bde173534d84e9336befc59d9c1fb505e6fd8e1c7ba9da0aca10\": container with ID starting with bd85ddf58d29bde173534d84e9336befc59d9c1fb505e6fd8e1c7ba9da0aca10 not found: ID does not exist"
Sep 17 03:44:33.561088 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.561026    2557 scope.go:117] "RemoveContainer" containerID="d3cc3ab52306b191727104f4d1392040c6fd411795a4a86250da2f3af299f728"
Sep 17 03:44:33.561582 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:44:33.561331    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3cc3ab52306b191727104f4d1392040c6fd411795a4a86250da2f3af299f728\": container with ID starting with d3cc3ab52306b191727104f4d1392040c6fd411795a4a86250da2f3af299f728 not found: ID does not exist" containerID="d3cc3ab52306b191727104f4d1392040c6fd411795a4a86250da2f3af299f728"
Sep 17 03:44:33.561582 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.561353    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3cc3ab52306b191727104f4d1392040c6fd411795a4a86250da2f3af299f728"} err="failed to get container status \"d3cc3ab52306b191727104f4d1392040c6fd411795a4a86250da2f3af299f728\": rpc error: code = NotFound desc = could not find container \"d3cc3ab52306b191727104f4d1392040c6fd411795a4a86250da2f3af299f728\": container with ID starting with d3cc3ab52306b191727104f4d1392040c6fd411795a4a86250da2f3af299f728 not found: ID does not exist"
Sep 17 03:44:33.561582 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.561368    2557 scope.go:117] "RemoveContainer" containerID="477c4d204afbd68035ce37cdedd4b008d7c57e1b4206512c238b824412d8467c"
Sep 17 03:44:33.574555 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.574372    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"]
Sep 17 03:44:33.585739 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.585709    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4a43bf-95c976b95-284rp"]
Sep 17 03:44:33.596872 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.596847    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"]
Sep 17 03:44:33.599464 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.599442    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1afd34-openshift-default-c4866984b-68bmf"]
Sep 17 03:44:33.608912 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.608886    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"]
Sep 17 03:44:33.616301 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.616280    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4a43bf-855fcf485f-f4vp4"]
Sep 17 03:44:33.915592 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.915562    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" path="/var/lib/kubelet/pods/4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94/volumes"
Sep 17 03:44:33.915997 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.915980    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55405e79-549b-49d5-9b5a-b9c8e3710cdb" path="/var/lib/kubelet/pods/55405e79-549b-49d5-9b5a-b9c8e3710cdb/volumes"
Sep 17 03:44:33.916369 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.916355    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae0183b3-188d-4f38-a159-18aa11c9bb29" path="/var/lib/kubelet/pods/ae0183b3-188d-4f38-a159-18aa11c9bb29/volumes"
Sep 17 03:44:33.916924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:33.916905    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1cb8456-384c-43d5-b904-d840ae827e59" path="/var/lib/kubelet/pods/b1cb8456-384c-43d5-b904-d840ae827e59/volumes"
Sep 17 03:44:38.634258 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.634223    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"]
Sep 17 03:44:38.635084 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.635029    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t" podUID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerName="istio-proxy" containerID="cri-o://89c1400cac61dd09ee7867a84f2db42379e1fe45b1ae5a28658cb04c3abcead1" gracePeriod=30
Sep 17 03:44:38.997002 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.996930    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49"]
Sep 17 03:44:38.997408 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997395    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4a8a698-79e4-48ad-984b-a1709305ff8c" containerName="main"
Sep 17 03:44:38.997408 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997409    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4a8a698-79e4-48ad-984b-a1709305ff8c" containerName="main"
Sep 17 03:44:38.997496 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997427    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae0183b3-188d-4f38-a159-18aa11c9bb29" containerName="maas-api"
Sep 17 03:44:38.997496 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997432    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae0183b3-188d-4f38-a159-18aa11c9bb29" containerName="maas-api"
Sep 17 03:44:38.997496 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997441    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerName="istio-proxy"
Sep 17 03:44:38.997496 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997447    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerName="istio-proxy"
Sep 17 03:44:38.997496 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997453    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerName="istio-proxy"
Sep 17 03:44:38.997496 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997458    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerName="istio-proxy"
Sep 17 03:44:38.997496 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997466    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1cb8456-384c-43d5-b904-d840ae827e59" containerName="payload-processing"
Sep 17 03:44:38.997496 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997471    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1cb8456-384c-43d5-b904-d840ae827e59" containerName="payload-processing"
Sep 17 03:44:38.997496 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997490    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55405e79-549b-49d5-9b5a-b9c8e3710cdb" containerName="revoke-keys"
Sep 17 03:44:38.997496 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997495    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="55405e79-549b-49d5-9b5a-b9c8e3710cdb" containerName="revoke-keys"
Sep 17 03:44:38.997843 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997554    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae0183b3-188d-4f38-a159-18aa11c9bb29" containerName="maas-api"
Sep 17 03:44:38.997843 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997563    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="55405e79-549b-49d5-9b5a-b9c8e3710cdb" containerName="revoke-keys"
Sep 17 03:44:38.997843 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997570    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1cb8456-384c-43d5-b904-d840ae827e59" containerName="payload-processing"
Sep 17 03:44:38.997843 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997575    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerName="istio-proxy"
Sep 17 03:44:38.997843 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:38.997584    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4a8a698-79e4-48ad-984b-a1709305ff8c" containerName="main"
Sep 17 03:44:39.002317 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:39.002300    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49"
Sep 17 03:44:39.004506 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:39.004484    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:44:39.007302 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:39.007281    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49"]
Sep 17 03:44:39.158493 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:39.158466    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4231adc8-3139-4dcc-b3e3-6cd26cbdf749-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49\" (UID: \"4231adc8-3139-4dcc-b3e3-6cd26cbdf749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49"
Sep 17 03:44:39.259997 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:39.259909    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4231adc8-3139-4dcc-b3e3-6cd26cbdf749-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49\" (UID: \"4231adc8-3139-4dcc-b3e3-6cd26cbdf749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49"
Sep 17 03:44:39.260557 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:39.260537    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4231adc8-3139-4dcc-b3e3-6cd26cbdf749-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49\" (UID: \"4231adc8-3139-4dcc-b3e3-6cd26cbdf749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49"
Sep 17 03:44:39.314735 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:39.314712    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49"
Sep 17 03:44:39.589956 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:39.589924    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49"]
Sep 17 03:44:39.591539 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:44:39.591513    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4231adc8_3139_4dcc_b3e3_6cd26cbdf749.slice/crio-647f6016a4dab3f727ff5e0b29525a894ab598022589e2d9aeba8a0725b6bb90 WatchSource:0}: Error finding container 647f6016a4dab3f727ff5e0b29525a894ab598022589e2d9aeba8a0725b6bb90: Status 404 returned error can't find the container with id 647f6016a4dab3f727ff5e0b29525a894ab598022589e2d9aeba8a0725b6bb90
Sep 17 03:44:40.579686 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:40.579655    2557 generic.go:356] "Generic (PLEG): container finished" podID="4231adc8-3139-4dcc-b3e3-6cd26cbdf749" containerID="e9a7104fe6711bdc61210e09d973b496a69d219ccd36ffc70dd9a7b1d49c385e" exitCode=0
Sep 17 03:44:40.580090 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:40.579732    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49" event={"ID":"4231adc8-3139-4dcc-b3e3-6cd26cbdf749","Type":"ContainerDied","Data":"e9a7104fe6711bdc61210e09d973b496a69d219ccd36ffc70dd9a7b1d49c385e"}
Sep 17 03:44:40.580090 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:40.579767    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49" event={"ID":"4231adc8-3139-4dcc-b3e3-6cd26cbdf749","Type":"ContainerStarted","Data":"647f6016a4dab3f727ff5e0b29525a894ab598022589e2d9aeba8a0725b6bb90"}
Sep 17 03:44:41.731963 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:41.731932    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49"
Sep 17 03:44:41.886375 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:41.886352    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4231adc8-3139-4dcc-b3e3-6cd26cbdf749-maas-api-service-ca\") pod \"4231adc8-3139-4dcc-b3e3-6cd26cbdf749\" (UID: \"4231adc8-3139-4dcc-b3e3-6cd26cbdf749\") "
Sep 17 03:44:41.886669 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:41.886649    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4231adc8-3139-4dcc-b3e3-6cd26cbdf749-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4231adc8-3139-4dcc-b3e3-6cd26cbdf749" (UID: "4231adc8-3139-4dcc-b3e3-6cd26cbdf749"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:44:41.987591 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:41.987566    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4231adc8-3139-4dcc-b3e3-6cd26cbdf749-maas-api-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:42.556770 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:42.556731    2557 patch_prober.go:28] interesting pod/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 03:44:42.556938 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:42.556798    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t" podUID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 03:44:42.602323 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:42.602293    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49"
Sep 17 03:44:42.602478 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:42.602293    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49" event={"ID":"4231adc8-3139-4dcc-b3e3-6cd26cbdf749","Type":"ContainerDied","Data":"647f6016a4dab3f727ff5e0b29525a894ab598022589e2d9aeba8a0725b6bb90"}
Sep 17 03:44:42.602478 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:42.602447    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="647f6016a4dab3f727ff5e0b29525a894ab598022589e2d9aeba8a0725b6bb90"
Sep 17 03:44:43.913783 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:43.913727    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:44:44.015990 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.015963    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-token\") pod \"ed111905-0ad6-4068-8e56-90df7fd45dc4\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") "
Sep 17 03:44:44.016135 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016060    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-workload-socket\") pod \"ed111905-0ad6-4068-8e56-90df7fd45dc4\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") "
Sep 17 03:44:44.016135 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016097    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed111905-0ad6-4068-8e56-90df7fd45dc4-istiod-ca-cert\") pod \"ed111905-0ad6-4068-8e56-90df7fd45dc4\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") "
Sep 17 03:44:44.016135 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016115    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-credential-socket\") pod \"ed111905-0ad6-4068-8e56-90df7fd45dc4\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") "
Sep 17 03:44:44.016325 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016158    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-podinfo\") pod \"ed111905-0ad6-4068-8e56-90df7fd45dc4\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") "
Sep 17 03:44:44.016325 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016188    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-workload-certs\") pod \"ed111905-0ad6-4068-8e56-90df7fd45dc4\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") "
Sep 17 03:44:44.016325 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016260    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-envoy\") pod \"ed111905-0ad6-4068-8e56-90df7fd45dc4\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") "
Sep 17 03:44:44.016325 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016288    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2xzt\" (UniqueName: \"kubernetes.io/projected/ed111905-0ad6-4068-8e56-90df7fd45dc4-kube-api-access-x2xzt\") pod \"ed111905-0ad6-4068-8e56-90df7fd45dc4\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") "
Sep 17 03:44:44.016540 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016335    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-data\") pod \"ed111905-0ad6-4068-8e56-90df7fd45dc4\" (UID: \"ed111905-0ad6-4068-8e56-90df7fd45dc4\") "
Sep 17 03:44:44.016540 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016430    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ed111905-0ad6-4068-8e56-90df7fd45dc4" (UID: "ed111905-0ad6-4068-8e56-90df7fd45dc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:44.016540 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016468    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ed111905-0ad6-4068-8e56-90df7fd45dc4" (UID: "ed111905-0ad6-4068-8e56-90df7fd45dc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:44.016708 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016614    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-data" (OuterVolumeSpecName: "istio-data") pod "ed111905-0ad6-4068-8e56-90df7fd45dc4" (UID: "ed111905-0ad6-4068-8e56-90df7fd45dc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:44.016708 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016656    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ed111905-0ad6-4068-8e56-90df7fd45dc4" (UID: "ed111905-0ad6-4068-8e56-90df7fd45dc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:44.016826 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016798    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:44.016826 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016806    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed111905-0ad6-4068-8e56-90df7fd45dc4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ed111905-0ad6-4068-8e56-90df7fd45dc4" (UID: "ed111905-0ad6-4068-8e56-90df7fd45dc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:44:44.016826 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016816    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:44.016992 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016841    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:44.016992 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.016856    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:44.018571 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.018528    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ed111905-0ad6-4068-8e56-90df7fd45dc4" (UID: "ed111905-0ad6-4068-8e56-90df7fd45dc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:44:44.018700 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.018678    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-token" (OuterVolumeSpecName: "istio-token") pod "ed111905-0ad6-4068-8e56-90df7fd45dc4" (UID: "ed111905-0ad6-4068-8e56-90df7fd45dc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:44:44.018854 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.018831    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ed111905-0ad6-4068-8e56-90df7fd45dc4" (UID: "ed111905-0ad6-4068-8e56-90df7fd45dc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:44:44.018974 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.018907    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed111905-0ad6-4068-8e56-90df7fd45dc4-kube-api-access-x2xzt" (OuterVolumeSpecName: "kube-api-access-x2xzt") pod "ed111905-0ad6-4068-8e56-90df7fd45dc4" (UID: "ed111905-0ad6-4068-8e56-90df7fd45dc4"). InnerVolumeSpecName "kube-api-access-x2xzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:44:44.117647 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.117618    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:44.117755 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.117652    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2xzt\" (UniqueName: \"kubernetes.io/projected/ed111905-0ad6-4068-8e56-90df7fd45dc4-kube-api-access-x2xzt\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:44.117755 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.117668    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:44.117755 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.117681    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed111905-0ad6-4068-8e56-90df7fd45dc4-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:44.117755 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.117693    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed111905-0ad6-4068-8e56-90df7fd45dc4-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:44:44.616433 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.616396    2557 generic.go:356] "Generic (PLEG): container finished" podID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerID="89c1400cac61dd09ee7867a84f2db42379e1fe45b1ae5a28658cb04c3abcead1" exitCode=0
Sep 17 03:44:44.616633 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.616444    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"
Sep 17 03:44:44.616633 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.616444    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t" event={"ID":"ed111905-0ad6-4068-8e56-90df7fd45dc4","Type":"ContainerDied","Data":"89c1400cac61dd09ee7867a84f2db42379e1fe45b1ae5a28658cb04c3abcead1"}
Sep 17 03:44:44.616633 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.616528    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t" event={"ID":"ed111905-0ad6-4068-8e56-90df7fd45dc4","Type":"ContainerDied","Data":"d6ab24d0b46eb4b13b5c50e59847494615fd8844eaf92ace7137a12f19b13baf"}
Sep 17 03:44:44.616633 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.616559    2557 scope.go:117] "RemoveContainer" containerID="89c1400cac61dd09ee7867a84f2db42379e1fe45b1ae5a28658cb04c3abcead1"
Sep 17 03:44:44.628152 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.628133    2557 scope.go:117] "RemoveContainer" containerID="71072c0d62247f01110f57d73cc030b83357b88c7fcac93a2dd201c2f613d43c"
Sep 17 03:44:44.636955 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.636930    2557 scope.go:117] "RemoveContainer" containerID="89c1400cac61dd09ee7867a84f2db42379e1fe45b1ae5a28658cb04c3abcead1"
Sep 17 03:44:44.637288 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:44:44.637261    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c1400cac61dd09ee7867a84f2db42379e1fe45b1ae5a28658cb04c3abcead1\": container with ID starting with 89c1400cac61dd09ee7867a84f2db42379e1fe45b1ae5a28658cb04c3abcead1 not found: ID does not exist" containerID="89c1400cac61dd09ee7867a84f2db42379e1fe45b1ae5a28658cb04c3abcead1"
Sep 17 03:44:44.637524 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.637298    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c1400cac61dd09ee7867a84f2db42379e1fe45b1ae5a28658cb04c3abcead1"} err="failed to get container status \"89c1400cac61dd09ee7867a84f2db42379e1fe45b1ae5a28658cb04c3abcead1\": rpc error: code = NotFound desc = could not find container \"89c1400cac61dd09ee7867a84f2db42379e1fe45b1ae5a28658cb04c3abcead1\": container with ID starting with 89c1400cac61dd09ee7867a84f2db42379e1fe45b1ae5a28658cb04c3abcead1 not found: ID does not exist"
Sep 17 03:44:44.637524 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.637321    2557 scope.go:117] "RemoveContainer" containerID="71072c0d62247f01110f57d73cc030b83357b88c7fcac93a2dd201c2f613d43c"
Sep 17 03:44:44.638129 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:44:44.638104    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71072c0d62247f01110f57d73cc030b83357b88c7fcac93a2dd201c2f613d43c\": container with ID starting with 71072c0d62247f01110f57d73cc030b83357b88c7fcac93a2dd201c2f613d43c not found: ID does not exist" containerID="71072c0d62247f01110f57d73cc030b83357b88c7fcac93a2dd201c2f613d43c"
Sep 17 03:44:44.638240 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.638136    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71072c0d62247f01110f57d73cc030b83357b88c7fcac93a2dd201c2f613d43c"} err="failed to get container status \"71072c0d62247f01110f57d73cc030b83357b88c7fcac93a2dd201c2f613d43c\": rpc error: code = NotFound desc = could not find container \"71072c0d62247f01110f57d73cc030b83357b88c7fcac93a2dd201c2f613d43c\": container with ID starting with 71072c0d62247f01110f57d73cc030b83357b88c7fcac93a2dd201c2f613d43c not found: ID does not exist"
Sep 17 03:44:44.643706 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.642619    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"]
Sep 17 03:44:44.649877 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:44.649814    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4a43bf-openshift-default-56c9f9496-pbf5t"]
Sep 17 03:44:45.915363 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:45.915324    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed111905-0ad6-4068-8e56-90df7fd45dc4" path="/var/lib/kubelet/pods/ed111905-0ad6-4068-8e56-90df7fd45dc4/volumes"
Sep 17 03:44:54.632647 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:54.632614    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49"]
Sep 17 03:44:54.636100 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:54.636070    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26d5d1-cbd49"]
Sep 17 03:44:55.914829 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:44:55.914793    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4231adc8-3139-4dcc-b3e3-6cd26cbdf749" path="/var/lib/kubelet/pods/4231adc8-3139-4dcc-b3e3-6cd26cbdf749/volumes"
Sep 17 03:45:00.132501 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.132467    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fc820e-29826945-x48hc"]
Sep 17 03:45:00.132910 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.132897    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerName="istio-proxy"
Sep 17 03:45:00.132959 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.132912    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerName="istio-proxy"
Sep 17 03:45:00.132959 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.132936    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerName="istio-proxy"
Sep 17 03:45:00.132959 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.132941    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerName="istio-proxy"
Sep 17 03:45:00.132959 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.132953    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4231adc8-3139-4dcc-b3e3-6cd26cbdf749" containerName="revoke-keys"
Sep 17 03:45:00.132959 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.132958    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4231adc8-3139-4dcc-b3e3-6cd26cbdf749" containerName="revoke-keys"
Sep 17 03:45:00.133116 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.133018    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc11b8f-129f-48f6-b2c8-fbc4b4ca1b94" containerName="istio-proxy"
Sep 17 03:45:00.133116 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.133026    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerName="istio-proxy"
Sep 17 03:45:00.133116 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.133032    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed111905-0ad6-4068-8e56-90df7fd45dc4" containerName="istio-proxy"
Sep 17 03:45:00.133116 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.133039    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4231adc8-3139-4dcc-b3e3-6cd26cbdf749" containerName="revoke-keys"
Sep 17 03:45:00.138034 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.138015    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-22e045-29826945-wnsd6"]
Sep 17 03:45:00.138159 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.138134    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fc820e-29826945-x48hc"
Sep 17 03:45:00.140377 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.140351    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8vwnw\""
Sep 17 03:45:00.141598 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.141576    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-22e045-29826945-wnsd6"
Sep 17 03:45:00.148234 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.148210    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fc820e-29826945-x48hc"]
Sep 17 03:45:00.151182 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.151160    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-22e045-29826945-wnsd6"]
Sep 17 03:45:00.152284 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.152265    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fc820e-29826945-x48hc"
Sep 17 03:45:00.161046 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.161026    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-22e045-29826945-wnsd6"
Sep 17 03:45:00.437810 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.437783    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fc820e-29826945-x48hc"]
Sep 17 03:45:00.439749 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:45:00.439722    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2b3f5c_e605_4049_829f_5ec2438677c7.slice/crio-8087332b80a90183f05316c9097cb8d1460d9829e95076489bd85d381b7aca60 WatchSource:0}: Error finding container 8087332b80a90183f05316c9097cb8d1460d9829e95076489bd85d381b7aca60: Status 404 returned error can't find the container with id 8087332b80a90183f05316c9097cb8d1460d9829e95076489bd85d381b7aca60
Sep 17 03:45:00.449549 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.449526    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-22e045-29826945-wnsd6"]
Sep 17 03:45:00.457609 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:45:00.457575    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3f2d1a_1fc8_4176_aaef_613d952479ad.slice/crio-96093a23411e717f1c0afca9ce9c07e7360dccc4efc2c6b6379defb68ae7dd75 WatchSource:0}: Error finding container 96093a23411e717f1c0afca9ce9c07e7360dccc4efc2c6b6379defb68ae7dd75: Status 404 returned error can't find the container with id 96093a23411e717f1c0afca9ce9c07e7360dccc4efc2c6b6379defb68ae7dd75
Sep 17 03:45:00.712485 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.712413    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fc820e-29826945-x48hc" event={"ID":"1a2b3f5c-e605-4049-829f-5ec2438677c7","Type":"ContainerStarted","Data":"8087332b80a90183f05316c9097cb8d1460d9829e95076489bd85d381b7aca60"}
Sep 17 03:45:00.713730 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:00.713704    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-22e045-29826945-wnsd6" event={"ID":"2a3f2d1a-1fc8-4176-aaef-613d952479ad","Type":"ContainerStarted","Data":"96093a23411e717f1c0afca9ce9c07e7360dccc4efc2c6b6379defb68ae7dd75"}
Sep 17 03:45:01.723803 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:01.723768    2557 generic.go:356] "Generic (PLEG): container finished" podID="1a2b3f5c-e605-4049-829f-5ec2438677c7" containerID="ca4b5433c32bbf4474f8845662878ec2ed943167a9697bd610d2a70582fabc4a" exitCode=0
Sep 17 03:45:01.724296 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:01.723863    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fc820e-29826945-x48hc" event={"ID":"1a2b3f5c-e605-4049-829f-5ec2438677c7","Type":"ContainerDied","Data":"ca4b5433c32bbf4474f8845662878ec2ed943167a9697bd610d2a70582fabc4a"}
Sep 17 03:45:01.726667 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:01.726633    2557 generic.go:356] "Generic (PLEG): container finished" podID="2a3f2d1a-1fc8-4176-aaef-613d952479ad" containerID="73ab69dc258feaf4a250cb9051806df652cc6ac3a5cef7f57b3ac305be0d659d" exitCode=0
Sep 17 03:45:01.726813 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:01.726685    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-22e045-29826945-wnsd6" event={"ID":"2a3f2d1a-1fc8-4176-aaef-613d952479ad","Type":"ContainerDied","Data":"73ab69dc258feaf4a250cb9051806df652cc6ac3a5cef7f57b3ac305be0d659d"}
Sep 17 03:45:02.906332 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:02.906303    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-22e045-29826945-wnsd6"
Sep 17 03:45:02.936873 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:02.936839    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fc820e-29826945-x48hc"
Sep 17 03:45:03.742990 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:03.742951    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fc820e-29826945-x48hc"
Sep 17 03:45:03.743213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:03.742989    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fc820e-29826945-x48hc" event={"ID":"1a2b3f5c-e605-4049-829f-5ec2438677c7","Type":"ContainerDied","Data":"8087332b80a90183f05316c9097cb8d1460d9829e95076489bd85d381b7aca60"}
Sep 17 03:45:03.743213 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:03.743047    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8087332b80a90183f05316c9097cb8d1460d9829e95076489bd85d381b7aca60"
Sep 17 03:45:03.744771 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:03.744748    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-22e045-29826945-wnsd6"
Sep 17 03:45:03.744902 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:03.744778    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-22e045-29826945-wnsd6" event={"ID":"2a3f2d1a-1fc8-4176-aaef-613d952479ad","Type":"ContainerDied","Data":"96093a23411e717f1c0afca9ce9c07e7360dccc4efc2c6b6379defb68ae7dd75"}
Sep 17 03:45:03.744902 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:03.744819    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96093a23411e717f1c0afca9ce9c07e7360dccc4efc2c6b6379defb68ae7dd75"
Sep 17 03:45:12.752353 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:12.752323    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"]
Sep 17 03:45:12.753097 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:12.753063    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v" podUID="9b01fd2c-9199-4288-b509-029da1d8b9a6" containerName="payload-processing" containerID="cri-o://088b2e147b311fe7f42f953940e30915a6e9fffd7ff9ea8696d92bd7b88c3e8e" gracePeriod=30
Sep 17 03:45:12.757165 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:12.757139    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"]
Sep 17 03:45:12.757411 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:12.757339    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls" podUID="0d11737d-b322-4bef-9c56-5d1bcb52d5cd" containerName="payload-pre-processing" containerID="cri-o://1cd2e383319b34d5d2b602320f3a9185cdca269f7f9ff2c14ac5182859ba4736" gracePeriod=30
Sep 17 03:45:13.085387 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.085285    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"
Sep 17 03:45:13.088423 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.088402    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"
Sep 17 03:45:13.182895 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.182862    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qf5rs\" (UniqueName: \"kubernetes.io/projected/0d11737d-b322-4bef-9c56-5d1bcb52d5cd-kube-api-access-qf5rs\") pod \"0d11737d-b322-4bef-9c56-5d1bcb52d5cd\" (UID: \"0d11737d-b322-4bef-9c56-5d1bcb52d5cd\") "
Sep 17 03:45:13.183070 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.182926    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b01fd2c-9199-4288-b509-029da1d8b9a6-plugins-config-volume\") pod \"9b01fd2c-9199-4288-b509-029da1d8b9a6\" (UID: \"9b01fd2c-9199-4288-b509-029da1d8b9a6\") "
Sep 17 03:45:13.183070 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.182966    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tn7d\" (UniqueName: \"kubernetes.io/projected/9b01fd2c-9199-4288-b509-029da1d8b9a6-kube-api-access-8tn7d\") pod \"9b01fd2c-9199-4288-b509-029da1d8b9a6\" (UID: \"9b01fd2c-9199-4288-b509-029da1d8b9a6\") "
Sep 17 03:45:13.183070 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.183011    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d11737d-b322-4bef-9c56-5d1bcb52d5cd-plugins-config-volume\") pod \"0d11737d-b322-4bef-9c56-5d1bcb52d5cd\" (UID: \"0d11737d-b322-4bef-9c56-5d1bcb52d5cd\") "
Sep 17 03:45:13.183389 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.183349    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b01fd2c-9199-4288-b509-029da1d8b9a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b01fd2c-9199-4288-b509-029da1d8b9a6" (UID: "9b01fd2c-9199-4288-b509-029da1d8b9a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:45:13.183488 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.183453    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d11737d-b322-4bef-9c56-5d1bcb52d5cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d11737d-b322-4bef-9c56-5d1bcb52d5cd" (UID: "0d11737d-b322-4bef-9c56-5d1bcb52d5cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:45:13.183617 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.183497    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b01fd2c-9199-4288-b509-029da1d8b9a6-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:13.185178 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.185154    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d11737d-b322-4bef-9c56-5d1bcb52d5cd-kube-api-access-qf5rs" (OuterVolumeSpecName: "kube-api-access-qf5rs") pod "0d11737d-b322-4bef-9c56-5d1bcb52d5cd" (UID: "0d11737d-b322-4bef-9c56-5d1bcb52d5cd"). InnerVolumeSpecName "kube-api-access-qf5rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:45:13.185493 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.185408    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b01fd2c-9199-4288-b509-029da1d8b9a6-kube-api-access-8tn7d" (OuterVolumeSpecName: "kube-api-access-8tn7d") pod "9b01fd2c-9199-4288-b509-029da1d8b9a6" (UID: "9b01fd2c-9199-4288-b509-029da1d8b9a6"). InnerVolumeSpecName "kube-api-access-8tn7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:45:13.285022 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.284994    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf5rs\" (UniqueName: \"kubernetes.io/projected/0d11737d-b322-4bef-9c56-5d1bcb52d5cd-kube-api-access-qf5rs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:13.285022 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.285018    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tn7d\" (UniqueName: \"kubernetes.io/projected/9b01fd2c-9199-4288-b509-029da1d8b9a6-kube-api-access-8tn7d\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:13.285177 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.285028    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d11737d-b322-4bef-9c56-5d1bcb52d5cd-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:13.807227 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.807181    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b01fd2c-9199-4288-b509-029da1d8b9a6" containerID="088b2e147b311fe7f42f953940e30915a6e9fffd7ff9ea8696d92bd7b88c3e8e" exitCode=0
Sep 17 03:45:13.807611 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.807245    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"
Sep 17 03:45:13.807611 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.807265    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v" event={"ID":"9b01fd2c-9199-4288-b509-029da1d8b9a6","Type":"ContainerDied","Data":"088b2e147b311fe7f42f953940e30915a6e9fffd7ff9ea8696d92bd7b88c3e8e"}
Sep 17 03:45:13.807611 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.807311    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v" event={"ID":"9b01fd2c-9199-4288-b509-029da1d8b9a6","Type":"ContainerDied","Data":"8ead26145772d49dc0183eb8f7cab34a0a36918419a4c398d74b80635b33534c"}
Sep 17 03:45:13.807611 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.807334    2557 scope.go:117] "RemoveContainer" containerID="088b2e147b311fe7f42f953940e30915a6e9fffd7ff9ea8696d92bd7b88c3e8e"
Sep 17 03:45:13.808861 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.808837    2557 generic.go:356] "Generic (PLEG): container finished" podID="0d11737d-b322-4bef-9c56-5d1bcb52d5cd" containerID="1cd2e383319b34d5d2b602320f3a9185cdca269f7f9ff2c14ac5182859ba4736" exitCode=0
Sep 17 03:45:13.808926 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.808887    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls" event={"ID":"0d11737d-b322-4bef-9c56-5d1bcb52d5cd","Type":"ContainerDied","Data":"1cd2e383319b34d5d2b602320f3a9185cdca269f7f9ff2c14ac5182859ba4736"}
Sep 17 03:45:13.808980 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.808926    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls" event={"ID":"0d11737d-b322-4bef-9c56-5d1bcb52d5cd","Type":"ContainerDied","Data":"6f5de775a879cb6e88d6e25a7e6a0ce818580a67031b36e7e8ebdd28a0656aec"}
Sep 17 03:45:13.808980 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.808879    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"
Sep 17 03:45:13.818844 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.818815    2557 scope.go:117] "RemoveContainer" containerID="088b2e147b311fe7f42f953940e30915a6e9fffd7ff9ea8696d92bd7b88c3e8e"
Sep 17 03:45:13.819183 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:45:13.819158    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088b2e147b311fe7f42f953940e30915a6e9fffd7ff9ea8696d92bd7b88c3e8e\": container with ID starting with 088b2e147b311fe7f42f953940e30915a6e9fffd7ff9ea8696d92bd7b88c3e8e not found: ID does not exist" containerID="088b2e147b311fe7f42f953940e30915a6e9fffd7ff9ea8696d92bd7b88c3e8e"
Sep 17 03:45:13.819295 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.819219    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088b2e147b311fe7f42f953940e30915a6e9fffd7ff9ea8696d92bd7b88c3e8e"} err="failed to get container status \"088b2e147b311fe7f42f953940e30915a6e9fffd7ff9ea8696d92bd7b88c3e8e\": rpc error: code = NotFound desc = could not find container \"088b2e147b311fe7f42f953940e30915a6e9fffd7ff9ea8696d92bd7b88c3e8e\": container with ID starting with 088b2e147b311fe7f42f953940e30915a6e9fffd7ff9ea8696d92bd7b88c3e8e not found: ID does not exist"
Sep 17 03:45:13.819295 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.819244    2557 scope.go:117] "RemoveContainer" containerID="1cd2e383319b34d5d2b602320f3a9185cdca269f7f9ff2c14ac5182859ba4736"
Sep 17 03:45:13.829321 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.829305    2557 scope.go:117] "RemoveContainer" containerID="1cd2e383319b34d5d2b602320f3a9185cdca269f7f9ff2c14ac5182859ba4736"
Sep 17 03:45:13.829549 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:45:13.829531    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd2e383319b34d5d2b602320f3a9185cdca269f7f9ff2c14ac5182859ba4736\": container with ID starting with 1cd2e383319b34d5d2b602320f3a9185cdca269f7f9ff2c14ac5182859ba4736 not found: ID does not exist" containerID="1cd2e383319b34d5d2b602320f3a9185cdca269f7f9ff2c14ac5182859ba4736"
Sep 17 03:45:13.829605 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.829558    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd2e383319b34d5d2b602320f3a9185cdca269f7f9ff2c14ac5182859ba4736"} err="failed to get container status \"1cd2e383319b34d5d2b602320f3a9185cdca269f7f9ff2c14ac5182859ba4736\": rpc error: code = NotFound desc = could not find container \"1cd2e383319b34d5d2b602320f3a9185cdca269f7f9ff2c14ac5182859ba4736\": container with ID starting with 1cd2e383319b34d5d2b602320f3a9185cdca269f7f9ff2c14ac5182859ba4736 not found: ID does not exist"
Sep 17 03:45:13.835999 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.835975    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"]
Sep 17 03:45:13.840601 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.840579    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-57958f-597fb44f66-vctls"]
Sep 17 03:45:13.854054 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.854033    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"]
Sep 17 03:45:13.857651 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.857621    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-57958f-55dc9cb78f-2fk6v"]
Sep 17 03:45:13.915204 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.915171    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d11737d-b322-4bef-9c56-5d1bcb52d5cd" path="/var/lib/kubelet/pods/0d11737d-b322-4bef-9c56-5d1bcb52d5cd/volumes"
Sep 17 03:45:13.915634 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:13.915613    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b01fd2c-9199-4288-b509-029da1d8b9a6" path="/var/lib/kubelet/pods/9b01fd2c-9199-4288-b509-029da1d8b9a6/volumes"
Sep 17 03:45:15.980402 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:15.980362    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"]
Sep 17 03:45:15.980833 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:15.980587    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw" podUID="d524126a-97d5-4888-ad42-d850ab79e8a5" containerName="maas-api" containerID="cri-o://9a148efdb0d1955b1ff1755e34fdd8cb182f2b2af77db58dbdb23425657a76eb" gracePeriod=30
Sep 17 03:45:16.242175 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.242120    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:45:16.314410 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.314373    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d524126a-97d5-4888-ad42-d850ab79e8a5-openshift-service-ca\") pod \"d524126a-97d5-4888-ad42-d850ab79e8a5\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") "
Sep 17 03:45:16.314584 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.314472    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6swk\" (UniqueName: \"kubernetes.io/projected/d524126a-97d5-4888-ad42-d850ab79e8a5-kube-api-access-q6swk\") pod \"d524126a-97d5-4888-ad42-d850ab79e8a5\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") "
Sep 17 03:45:16.314584 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.314536    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d524126a-97d5-4888-ad42-d850ab79e8a5-metrics-certs\") pod \"d524126a-97d5-4888-ad42-d850ab79e8a5\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") "
Sep 17 03:45:16.314709 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.314620    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d524126a-97d5-4888-ad42-d850ab79e8a5-maas-api-tls\") pod \"d524126a-97d5-4888-ad42-d850ab79e8a5\" (UID: \"d524126a-97d5-4888-ad42-d850ab79e8a5\") "
Sep 17 03:45:16.315141 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.315113    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d524126a-97d5-4888-ad42-d850ab79e8a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d524126a-97d5-4888-ad42-d850ab79e8a5" (UID: "d524126a-97d5-4888-ad42-d850ab79e8a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:45:16.317425 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.317373    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d524126a-97d5-4888-ad42-d850ab79e8a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d524126a-97d5-4888-ad42-d850ab79e8a5" (UID: "d524126a-97d5-4888-ad42-d850ab79e8a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:45:16.317828 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.317787    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d524126a-97d5-4888-ad42-d850ab79e8a5-kube-api-access-q6swk" (OuterVolumeSpecName: "kube-api-access-q6swk") pod "d524126a-97d5-4888-ad42-d850ab79e8a5" (UID: "d524126a-97d5-4888-ad42-d850ab79e8a5"). InnerVolumeSpecName "kube-api-access-q6swk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:45:16.318030 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.318007    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d524126a-97d5-4888-ad42-d850ab79e8a5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d524126a-97d5-4888-ad42-d850ab79e8a5" (UID: "d524126a-97d5-4888-ad42-d850ab79e8a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:45:16.416123 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.416089    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d524126a-97d5-4888-ad42-d850ab79e8a5-maas-api-tls\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:16.416123 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.416118    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d524126a-97d5-4888-ad42-d850ab79e8a5-openshift-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:16.416123 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.416128    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6swk\" (UniqueName: \"kubernetes.io/projected/d524126a-97d5-4888-ad42-d850ab79e8a5-kube-api-access-q6swk\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:16.416394 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.416137    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d524126a-97d5-4888-ad42-d850ab79e8a5-metrics-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:16.830407 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.830372    2557 generic.go:356] "Generic (PLEG): container finished" podID="d524126a-97d5-4888-ad42-d850ab79e8a5" containerID="9a148efdb0d1955b1ff1755e34fdd8cb182f2b2af77db58dbdb23425657a76eb" exitCode=0
Sep 17 03:45:16.830570 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.830416    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"
Sep 17 03:45:16.830570 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.830470    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw" event={"ID":"d524126a-97d5-4888-ad42-d850ab79e8a5","Type":"ContainerDied","Data":"9a148efdb0d1955b1ff1755e34fdd8cb182f2b2af77db58dbdb23425657a76eb"}
Sep 17 03:45:16.830570 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.830522    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw" event={"ID":"d524126a-97d5-4888-ad42-d850ab79e8a5","Type":"ContainerDied","Data":"4b8bd17f1ad248bdbb6ec53200b76aa4e09b9d097c8fea5ed1281fd1f0009099"}
Sep 17 03:45:16.830570 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.830555    2557 scope.go:117] "RemoveContainer" containerID="9a148efdb0d1955b1ff1755e34fdd8cb182f2b2af77db58dbdb23425657a76eb"
Sep 17 03:45:16.840436 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.840420    2557 scope.go:117] "RemoveContainer" containerID="9a148efdb0d1955b1ff1755e34fdd8cb182f2b2af77db58dbdb23425657a76eb"
Sep 17 03:45:16.840699 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:45:16.840682    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a148efdb0d1955b1ff1755e34fdd8cb182f2b2af77db58dbdb23425657a76eb\": container with ID starting with 9a148efdb0d1955b1ff1755e34fdd8cb182f2b2af77db58dbdb23425657a76eb not found: ID does not exist" containerID="9a148efdb0d1955b1ff1755e34fdd8cb182f2b2af77db58dbdb23425657a76eb"
Sep 17 03:45:16.840752 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.840708    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a148efdb0d1955b1ff1755e34fdd8cb182f2b2af77db58dbdb23425657a76eb"} err="failed to get container status \"9a148efdb0d1955b1ff1755e34fdd8cb182f2b2af77db58dbdb23425657a76eb\": rpc error: code = NotFound desc = could not find container \"9a148efdb0d1955b1ff1755e34fdd8cb182f2b2af77db58dbdb23425657a76eb\": container with ID starting with 9a148efdb0d1955b1ff1755e34fdd8cb182f2b2af77db58dbdb23425657a76eb not found: ID does not exist"
Sep 17 03:45:16.851064 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.851042    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"]
Sep 17 03:45:16.854996 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:16.854975    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-214fea-6cf4968bcd-5ctqw"]
Sep 17 03:45:17.915255 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:17.915227    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d524126a-97d5-4888-ad42-d850ab79e8a5" path="/var/lib/kubelet/pods/d524126a-97d5-4888-ad42-d850ab79e8a5/volumes"
Sep 17 03:45:19.937214 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:19.929924    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"]
Sep 17 03:45:19.937214 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:19.930257    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm" podUID="4a823a95-f61d-40c9-aaca-238000c3e77e" containerName="istio-proxy" containerID="cri-o://ca1182adc906a4aa5c29a20d431b591b7399c2c35f300a23737117b2874193e9" gracePeriod=30
Sep 17 03:45:22.877024 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:22.876991    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"]
Sep 17 03:45:22.877488 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:22.877218    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74" podUID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerName="istio-proxy" containerID="cri-o://ee8fa69d8eb077b01bbb902a26de07cf5f2ef01652a2cc75a21aef02a649b411" gracePeriod=30
Sep 17 03:45:25.180314 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.180291    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:45:25.302831 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.302769    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-credential-socket\") pod \"4a823a95-f61d-40c9-aaca-238000c3e77e\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") "
Sep 17 03:45:25.302831 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.302804    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9d4t\" (UniqueName: \"kubernetes.io/projected/4a823a95-f61d-40c9-aaca-238000c3e77e-kube-api-access-r9d4t\") pod \"4a823a95-f61d-40c9-aaca-238000c3e77e\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") "
Sep 17 03:45:25.303016 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.302842    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-workload-socket\") pod \"4a823a95-f61d-40c9-aaca-238000c3e77e\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") "
Sep 17 03:45:25.303016 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.302865    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-data\") pod \"4a823a95-f61d-40c9-aaca-238000c3e77e\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") "
Sep 17 03:45:25.303016 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.302892    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-envoy\") pod \"4a823a95-f61d-40c9-aaca-238000c3e77e\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") "
Sep 17 03:45:25.303016 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.302933    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-workload-certs\") pod \"4a823a95-f61d-40c9-aaca-238000c3e77e\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") "
Sep 17 03:45:25.303016 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.302973    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-token\") pod \"4a823a95-f61d-40c9-aaca-238000c3e77e\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") "
Sep 17 03:45:25.303016 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.303001    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-podinfo\") pod \"4a823a95-f61d-40c9-aaca-238000c3e77e\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") "
Sep 17 03:45:25.303347 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.303048    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a823a95-f61d-40c9-aaca-238000c3e77e-istiod-ca-cert\") pod \"4a823a95-f61d-40c9-aaca-238000c3e77e\" (UID: \"4a823a95-f61d-40c9-aaca-238000c3e77e\") "
Sep 17 03:45:25.303347 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.303082    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4a823a95-f61d-40c9-aaca-238000c3e77e" (UID: "4a823a95-f61d-40c9-aaca-238000c3e77e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:25.303347 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.303222    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-data" (OuterVolumeSpecName: "istio-data") pod "4a823a95-f61d-40c9-aaca-238000c3e77e" (UID: "4a823a95-f61d-40c9-aaca-238000c3e77e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:25.303347 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.303228    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4a823a95-f61d-40c9-aaca-238000c3e77e" (UID: "4a823a95-f61d-40c9-aaca-238000c3e77e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:25.303559 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.303469    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4a823a95-f61d-40c9-aaca-238000c3e77e" (UID: "4a823a95-f61d-40c9-aaca-238000c3e77e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:25.303559 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.303482    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a823a95-f61d-40c9-aaca-238000c3e77e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4a823a95-f61d-40c9-aaca-238000c3e77e" (UID: "4a823a95-f61d-40c9-aaca-238000c3e77e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:45:25.303685 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.303594    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a823a95-f61d-40c9-aaca-238000c3e77e-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:25.303685 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.303616    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:25.303685 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.303647    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:25.303685 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.303660    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:25.303685 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.303674    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:25.305074 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.305042    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a823a95-f61d-40c9-aaca-238000c3e77e-kube-api-access-r9d4t" (OuterVolumeSpecName: "kube-api-access-r9d4t") pod "4a823a95-f61d-40c9-aaca-238000c3e77e" (UID: "4a823a95-f61d-40c9-aaca-238000c3e77e"). InnerVolumeSpecName "kube-api-access-r9d4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:45:25.305309 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.305237    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-token" (OuterVolumeSpecName: "istio-token") pod "4a823a95-f61d-40c9-aaca-238000c3e77e" (UID: "4a823a95-f61d-40c9-aaca-238000c3e77e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:45:25.305801 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.305772    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4a823a95-f61d-40c9-aaca-238000c3e77e" (UID: "4a823a95-f61d-40c9-aaca-238000c3e77e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:25.305884 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.305844    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4a823a95-f61d-40c9-aaca-238000c3e77e" (UID: "4a823a95-f61d-40c9-aaca-238000c3e77e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:45:25.404840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.404819    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:25.404840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.404839    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:25.404970 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.404848    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a823a95-f61d-40c9-aaca-238000c3e77e-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:25.404970 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.404856    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9d4t\" (UniqueName: \"kubernetes.io/projected/4a823a95-f61d-40c9-aaca-238000c3e77e-kube-api-access-r9d4t\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:25.886704 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.886672    2557 generic.go:356] "Generic (PLEG): container finished" podID="4a823a95-f61d-40c9-aaca-238000c3e77e" containerID="ca1182adc906a4aa5c29a20d431b591b7399c2c35f300a23737117b2874193e9" exitCode=0
Sep 17 03:45:25.886941 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.886720    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"
Sep 17 03:45:25.886941 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.886749    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm" event={"ID":"4a823a95-f61d-40c9-aaca-238000c3e77e","Type":"ContainerDied","Data":"ca1182adc906a4aa5c29a20d431b591b7399c2c35f300a23737117b2874193e9"}
Sep 17 03:45:25.886941 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.886802    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm" event={"ID":"4a823a95-f61d-40c9-aaca-238000c3e77e","Type":"ContainerDied","Data":"d6ee543b7ba3b2d584e1a09c7d0eee66d565f47865cf5f12462e1de44b108322"}
Sep 17 03:45:25.886941 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.886824    2557 scope.go:117] "RemoveContainer" containerID="ca1182adc906a4aa5c29a20d431b591b7399c2c35f300a23737117b2874193e9"
Sep 17 03:45:25.896536 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.896516    2557 scope.go:117] "RemoveContainer" containerID="5e31008dc9ce71df6f50cff759cb26eea9ca0e1d35612e0bc0ce6225b300a4bf"
Sep 17 03:45:25.904429 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.904412    2557 scope.go:117] "RemoveContainer" containerID="ca1182adc906a4aa5c29a20d431b591b7399c2c35f300a23737117b2874193e9"
Sep 17 03:45:25.904694 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:45:25.904676    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca1182adc906a4aa5c29a20d431b591b7399c2c35f300a23737117b2874193e9\": container with ID starting with ca1182adc906a4aa5c29a20d431b591b7399c2c35f300a23737117b2874193e9 not found: ID does not exist" containerID="ca1182adc906a4aa5c29a20d431b591b7399c2c35f300a23737117b2874193e9"
Sep 17 03:45:25.904740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.904702    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca1182adc906a4aa5c29a20d431b591b7399c2c35f300a23737117b2874193e9"} err="failed to get container status \"ca1182adc906a4aa5c29a20d431b591b7399c2c35f300a23737117b2874193e9\": rpc error: code = NotFound desc = could not find container \"ca1182adc906a4aa5c29a20d431b591b7399c2c35f300a23737117b2874193e9\": container with ID starting with ca1182adc906a4aa5c29a20d431b591b7399c2c35f300a23737117b2874193e9 not found: ID does not exist"
Sep 17 03:45:25.904740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.904719    2557 scope.go:117] "RemoveContainer" containerID="5e31008dc9ce71df6f50cff759cb26eea9ca0e1d35612e0bc0ce6225b300a4bf"
Sep 17 03:45:25.904946 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:45:25.904930    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e31008dc9ce71df6f50cff759cb26eea9ca0e1d35612e0bc0ce6225b300a4bf\": container with ID starting with 5e31008dc9ce71df6f50cff759cb26eea9ca0e1d35612e0bc0ce6225b300a4bf not found: ID does not exist" containerID="5e31008dc9ce71df6f50cff759cb26eea9ca0e1d35612e0bc0ce6225b300a4bf"
Sep 17 03:45:25.904985 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.904952    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e31008dc9ce71df6f50cff759cb26eea9ca0e1d35612e0bc0ce6225b300a4bf"} err="failed to get container status \"5e31008dc9ce71df6f50cff759cb26eea9ca0e1d35612e0bc0ce6225b300a4bf\": rpc error: code = NotFound desc = could not find container \"5e31008dc9ce71df6f50cff759cb26eea9ca0e1d35612e0bc0ce6225b300a4bf\": container with ID starting with 5e31008dc9ce71df6f50cff759cb26eea9ca0e1d35612e0bc0ce6225b300a4bf not found: ID does not exist"
Sep 17 03:45:25.915536 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.915519    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"]
Sep 17 03:45:25.915620 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:25.915547    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-57958f-openshift-default-6544794f9d-8tkpm"]
Sep 17 03:45:27.914589 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:27.914559    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a823a95-f61d-40c9-aaca-238000c3e77e" path="/var/lib/kubelet/pods/4a823a95-f61d-40c9-aaca-238000c3e77e/volumes"
Sep 17 03:45:28.131459 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.131431    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:45:28.229926 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.229862    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-envoy\") pod \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") "
Sep 17 03:45:28.229926 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.229900    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-credential-socket\") pod \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") "
Sep 17 03:45:28.230133 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.229960    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx2lh\" (UniqueName: \"kubernetes.io/projected/740bc52a-9b17-4fe7-ba69-fe88ab56706d-kube-api-access-jx2lh\") pod \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") "
Sep 17 03:45:28.230133 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.229978    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istiod-ca-cert\") pod \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") "
Sep 17 03:45:28.230133 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.229994    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-podinfo\") pod \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") "
Sep 17 03:45:28.230133 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.230016    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-token\") pod \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") "
Sep 17 03:45:28.230133 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.230049    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-data\") pod \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") "
Sep 17 03:45:28.230133 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.230101    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-workload-socket\") pod \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") "
Sep 17 03:45:28.230133 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.230134    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-workload-certs\") pod \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\" (UID: \"740bc52a-9b17-4fe7-ba69-fe88ab56706d\") "
Sep 17 03:45:28.230547 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.230389    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "740bc52a-9b17-4fe7-ba69-fe88ab56706d" (UID: "740bc52a-9b17-4fe7-ba69-fe88ab56706d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:45:28.230547 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.230403    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-data" (OuterVolumeSpecName: "istio-data") pod "740bc52a-9b17-4fe7-ba69-fe88ab56706d" (UID: "740bc52a-9b17-4fe7-ba69-fe88ab56706d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:28.230656 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.230585    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "740bc52a-9b17-4fe7-ba69-fe88ab56706d" (UID: "740bc52a-9b17-4fe7-ba69-fe88ab56706d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:28.230740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.230672    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "740bc52a-9b17-4fe7-ba69-fe88ab56706d" (UID: "740bc52a-9b17-4fe7-ba69-fe88ab56706d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:28.230740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.230687    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:28.230740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.230705    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:28.230740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.230717    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:28.230955 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.230763    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "740bc52a-9b17-4fe7-ba69-fe88ab56706d" (UID: "740bc52a-9b17-4fe7-ba69-fe88ab56706d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:28.232299 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.232276    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740bc52a-9b17-4fe7-ba69-fe88ab56706d-kube-api-access-jx2lh" (OuterVolumeSpecName: "kube-api-access-jx2lh") pod "740bc52a-9b17-4fe7-ba69-fe88ab56706d" (UID: "740bc52a-9b17-4fe7-ba69-fe88ab56706d"). InnerVolumeSpecName "kube-api-access-jx2lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:45:28.232467 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.232450    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-token" (OuterVolumeSpecName: "istio-token") pod "740bc52a-9b17-4fe7-ba69-fe88ab56706d" (UID: "740bc52a-9b17-4fe7-ba69-fe88ab56706d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:45:28.232589 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.232570    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "740bc52a-9b17-4fe7-ba69-fe88ab56706d" (UID: "740bc52a-9b17-4fe7-ba69-fe88ab56706d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:28.232650 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.232574    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "740bc52a-9b17-4fe7-ba69-fe88ab56706d" (UID: "740bc52a-9b17-4fe7-ba69-fe88ab56706d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:45:28.331241 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.331214    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx2lh\" (UniqueName: \"kubernetes.io/projected/740bc52a-9b17-4fe7-ba69-fe88ab56706d-kube-api-access-jx2lh\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:28.331241 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.331236    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:28.331380 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.331245    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:28.331380 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.331254    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:28.331380 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.331263    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:28.331380 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.331271    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740bc52a-9b17-4fe7-ba69-fe88ab56706d-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:28.907533 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.907506    2557 generic.go:356] "Generic (PLEG): container finished" podID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerID="ee8fa69d8eb077b01bbb902a26de07cf5f2ef01652a2cc75a21aef02a649b411" exitCode=0
Sep 17 03:45:28.907661 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.907551    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"
Sep 17 03:45:28.907661 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.907549    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74" event={"ID":"740bc52a-9b17-4fe7-ba69-fe88ab56706d","Type":"ContainerDied","Data":"ee8fa69d8eb077b01bbb902a26de07cf5f2ef01652a2cc75a21aef02a649b411"}
Sep 17 03:45:28.907746 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.907682    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74" event={"ID":"740bc52a-9b17-4fe7-ba69-fe88ab56706d","Type":"ContainerDied","Data":"e415e73ca731d1378bf2b2b7e0bb4866cb70d036ea3c6dce762a98f79eabb22c"}
Sep 17 03:45:28.907746 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.907711    2557 scope.go:117] "RemoveContainer" containerID="ee8fa69d8eb077b01bbb902a26de07cf5f2ef01652a2cc75a21aef02a649b411"
Sep 17 03:45:28.917736 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.917690    2557 scope.go:117] "RemoveContainer" containerID="db2d98eba7405344e57693995ae8b20ea5366c2e278932cecce4baf9b194eec2"
Sep 17 03:45:28.925850 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.925827    2557 scope.go:117] "RemoveContainer" containerID="ee8fa69d8eb077b01bbb902a26de07cf5f2ef01652a2cc75a21aef02a649b411"
Sep 17 03:45:28.926473 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:45:28.926450    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee8fa69d8eb077b01bbb902a26de07cf5f2ef01652a2cc75a21aef02a649b411\": container with ID starting with ee8fa69d8eb077b01bbb902a26de07cf5f2ef01652a2cc75a21aef02a649b411 not found: ID does not exist" containerID="ee8fa69d8eb077b01bbb902a26de07cf5f2ef01652a2cc75a21aef02a649b411"
Sep 17 03:45:28.926634 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.926600    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee8fa69d8eb077b01bbb902a26de07cf5f2ef01652a2cc75a21aef02a649b411"} err="failed to get container status \"ee8fa69d8eb077b01bbb902a26de07cf5f2ef01652a2cc75a21aef02a649b411\": rpc error: code = NotFound desc = could not find container \"ee8fa69d8eb077b01bbb902a26de07cf5f2ef01652a2cc75a21aef02a649b411\": container with ID starting with ee8fa69d8eb077b01bbb902a26de07cf5f2ef01652a2cc75a21aef02a649b411 not found: ID does not exist"
Sep 17 03:45:28.926634 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.926634    2557 scope.go:117] "RemoveContainer" containerID="db2d98eba7405344e57693995ae8b20ea5366c2e278932cecce4baf9b194eec2"
Sep 17 03:45:28.926937 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:45:28.926916    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db2d98eba7405344e57693995ae8b20ea5366c2e278932cecce4baf9b194eec2\": container with ID starting with db2d98eba7405344e57693995ae8b20ea5366c2e278932cecce4baf9b194eec2 not found: ID does not exist" containerID="db2d98eba7405344e57693995ae8b20ea5366c2e278932cecce4baf9b194eec2"
Sep 17 03:45:28.926990 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.926942    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2d98eba7405344e57693995ae8b20ea5366c2e278932cecce4baf9b194eec2"} err="failed to get container status \"db2d98eba7405344e57693995ae8b20ea5366c2e278932cecce4baf9b194eec2\": rpc error: code = NotFound desc = could not find container \"db2d98eba7405344e57693995ae8b20ea5366c2e278932cecce4baf9b194eec2\": container with ID starting with db2d98eba7405344e57693995ae8b20ea5366c2e278932cecce4baf9b194eec2 not found: ID does not exist"
Sep 17 03:45:28.928811 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.928792    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"]
Sep 17 03:45:28.933740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:28.933719    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-214fea-openshift-default-6cb4f7d54b-w8c74"]
Sep 17 03:45:29.915790 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:29.915758    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" path="/var/lib/kubelet/pods/740bc52a-9b17-4fe7-ba69-fe88ab56706d/volumes"
Sep 17 03:45:30.781665 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:30.781622    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fc820e-29826945-x48hc"]
Sep 17 03:45:30.784070 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:30.784046    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fc820e-29826945-x48hc"]
Sep 17 03:45:31.917772 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:31.917738    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a2b3f5c-e605-4049-829f-5ec2438677c7" path="/var/lib/kubelet/pods/1a2b3f5c-e605-4049-829f-5ec2438677c7/volumes"
Sep 17 03:45:37.382136 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:37.382103    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"]
Sep 17 03:45:37.382652 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:37.382368    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n" podUID="4122da61-e79e-434b-83df-15053d1edeb5" containerName="istio-proxy" containerID="cri-o://4ba83ac674dae741d583ef45ac5d2b140a33c52b738310a55c0e37fcd42a5422" gracePeriod=30
Sep 17 03:45:42.647171 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.647115    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:45:42.746612 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.746580    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-workload-socket\") pod \"4122da61-e79e-434b-83df-15053d1edeb5\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") "
Sep 17 03:45:42.746759 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.746629    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4122da61-e79e-434b-83df-15053d1edeb5-istio-podinfo\") pod \"4122da61-e79e-434b-83df-15053d1edeb5\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") "
Sep 17 03:45:42.746759 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.746670    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-istio-data\") pod \"4122da61-e79e-434b-83df-15053d1edeb5\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") "
Sep 17 03:45:42.746759 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.746692    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4122da61-e79e-434b-83df-15053d1edeb5-istiod-ca-cert\") pod \"4122da61-e79e-434b-83df-15053d1edeb5\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") "
Sep 17 03:45:42.746759 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.746745    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-credential-socket\") pod \"4122da61-e79e-434b-83df-15053d1edeb5\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") "
Sep 17 03:45:42.746998 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.746815    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dhq2\" (UniqueName: \"kubernetes.io/projected/4122da61-e79e-434b-83df-15053d1edeb5-kube-api-access-7dhq2\") pod \"4122da61-e79e-434b-83df-15053d1edeb5\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") "
Sep 17 03:45:42.746998 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.746859    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4122da61-e79e-434b-83df-15053d1edeb5-istio-token\") pod \"4122da61-e79e-434b-83df-15053d1edeb5\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") "
Sep 17 03:45:42.746998 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.746886    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-workload-certs\") pod \"4122da61-e79e-434b-83df-15053d1edeb5\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") "
Sep 17 03:45:42.746998 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.746890    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4122da61-e79e-434b-83df-15053d1edeb5" (UID: "4122da61-e79e-434b-83df-15053d1edeb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:42.746998 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.746913    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-istio-envoy\") pod \"4122da61-e79e-434b-83df-15053d1edeb5\" (UID: \"4122da61-e79e-434b-83df-15053d1edeb5\") "
Sep 17 03:45:42.747298 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.747042    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-istio-data" (OuterVolumeSpecName: "istio-data") pod "4122da61-e79e-434b-83df-15053d1edeb5" (UID: "4122da61-e79e-434b-83df-15053d1edeb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:42.747298 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.747168    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4122da61-e79e-434b-83df-15053d1edeb5" (UID: "4122da61-e79e-434b-83df-15053d1edeb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:42.747298 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.747227    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4122da61-e79e-434b-83df-15053d1edeb5" (UID: "4122da61-e79e-434b-83df-15053d1edeb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:42.747298 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.747286    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4122da61-e79e-434b-83df-15053d1edeb5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4122da61-e79e-434b-83df-15053d1edeb5" (UID: "4122da61-e79e-434b-83df-15053d1edeb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:45:42.747517 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.747342    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:42.747517 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.747359    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:42.747517 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.747372    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:42.747517 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.747386    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:42.748947 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.748922    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4122da61-e79e-434b-83df-15053d1edeb5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4122da61-e79e-434b-83df-15053d1edeb5" (UID: "4122da61-e79e-434b-83df-15053d1edeb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:45:42.749138 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.749118    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4122da61-e79e-434b-83df-15053d1edeb5-kube-api-access-7dhq2" (OuterVolumeSpecName: "kube-api-access-7dhq2") pod "4122da61-e79e-434b-83df-15053d1edeb5" (UID: "4122da61-e79e-434b-83df-15053d1edeb5"). InnerVolumeSpecName "kube-api-access-7dhq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:45:42.749555 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.749537    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4122da61-e79e-434b-83df-15053d1edeb5" (UID: "4122da61-e79e-434b-83df-15053d1edeb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:45:42.749555 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.749547    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4122da61-e79e-434b-83df-15053d1edeb5-istio-token" (OuterVolumeSpecName: "istio-token") pod "4122da61-e79e-434b-83df-15053d1edeb5" (UID: "4122da61-e79e-434b-83df-15053d1edeb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:45:42.847820 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.847796    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4122da61-e79e-434b-83df-15053d1edeb5-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:42.847820 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.847818    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dhq2\" (UniqueName: \"kubernetes.io/projected/4122da61-e79e-434b-83df-15053d1edeb5-kube-api-access-7dhq2\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:42.847968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.847828    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4122da61-e79e-434b-83df-15053d1edeb5-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:42.847968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.847836    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4122da61-e79e-434b-83df-15053d1edeb5-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:42.847968 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.847844    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4122da61-e79e-434b-83df-15053d1edeb5-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:45:42.997233 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.997157    2557 generic.go:356] "Generic (PLEG): container finished" podID="4122da61-e79e-434b-83df-15053d1edeb5" containerID="4ba83ac674dae741d583ef45ac5d2b140a33c52b738310a55c0e37fcd42a5422" exitCode=0
Sep 17 03:45:42.997233 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.997225    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"
Sep 17 03:45:42.997377 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.997225    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n" event={"ID":"4122da61-e79e-434b-83df-15053d1edeb5","Type":"ContainerDied","Data":"4ba83ac674dae741d583ef45ac5d2b140a33c52b738310a55c0e37fcd42a5422"}
Sep 17 03:45:42.997377 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.997339    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n" event={"ID":"4122da61-e79e-434b-83df-15053d1edeb5","Type":"ContainerDied","Data":"54eb8e29663b7802fdee5e1cde71efcce4bbe0e12aad5b817e9b53e2c29c2213"}
Sep 17 03:45:42.997377 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:42.997365    2557 scope.go:117] "RemoveContainer" containerID="4ba83ac674dae741d583ef45ac5d2b140a33c52b738310a55c0e37fcd42a5422"
Sep 17 03:45:43.008545 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:43.008517    2557 scope.go:117] "RemoveContainer" containerID="a0eb1fb74bb2642657d3339d887b15ca0033e6928068492ccfa2ba9e56c79e0a"
Sep 17 03:45:43.017075 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:43.017057    2557 scope.go:117] "RemoveContainer" containerID="4ba83ac674dae741d583ef45ac5d2b140a33c52b738310a55c0e37fcd42a5422"
Sep 17 03:45:43.017424 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:45:43.017398    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba83ac674dae741d583ef45ac5d2b140a33c52b738310a55c0e37fcd42a5422\": container with ID starting with 4ba83ac674dae741d583ef45ac5d2b140a33c52b738310a55c0e37fcd42a5422 not found: ID does not exist" containerID="4ba83ac674dae741d583ef45ac5d2b140a33c52b738310a55c0e37fcd42a5422"
Sep 17 03:45:43.017526 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:43.017432    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba83ac674dae741d583ef45ac5d2b140a33c52b738310a55c0e37fcd42a5422"} err="failed to get container status \"4ba83ac674dae741d583ef45ac5d2b140a33c52b738310a55c0e37fcd42a5422\": rpc error: code = NotFound desc = could not find container \"4ba83ac674dae741d583ef45ac5d2b140a33c52b738310a55c0e37fcd42a5422\": container with ID starting with 4ba83ac674dae741d583ef45ac5d2b140a33c52b738310a55c0e37fcd42a5422 not found: ID does not exist"
Sep 17 03:45:43.017526 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:43.017451    2557 scope.go:117] "RemoveContainer" containerID="a0eb1fb74bb2642657d3339d887b15ca0033e6928068492ccfa2ba9e56c79e0a"
Sep 17 03:45:43.017708 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:45:43.017688    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0eb1fb74bb2642657d3339d887b15ca0033e6928068492ccfa2ba9e56c79e0a\": container with ID starting with a0eb1fb74bb2642657d3339d887b15ca0033e6928068492ccfa2ba9e56c79e0a not found: ID does not exist" containerID="a0eb1fb74bb2642657d3339d887b15ca0033e6928068492ccfa2ba9e56c79e0a"
Sep 17 03:45:43.017846 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:43.017713    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0eb1fb74bb2642657d3339d887b15ca0033e6928068492ccfa2ba9e56c79e0a"} err="failed to get container status \"a0eb1fb74bb2642657d3339d887b15ca0033e6928068492ccfa2ba9e56c79e0a\": rpc error: code = NotFound desc = could not find container \"a0eb1fb74bb2642657d3339d887b15ca0033e6928068492ccfa2ba9e56c79e0a\": container with ID starting with a0eb1fb74bb2642657d3339d887b15ca0033e6928068492ccfa2ba9e56c79e0a not found: ID does not exist"
Sep 17 03:45:43.019168 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:43.019150    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"]
Sep 17 03:45:43.023544 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:43.023520    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fc820e-openshift-default-6fd77df564-zsr9n"]
Sep 17 03:45:43.916154 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:45:43.916120    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4122da61-e79e-434b-83df-15053d1edeb5" path="/var/lib/kubelet/pods/4122da61-e79e-434b-83df-15053d1edeb5/volumes"
Sep 17 03:47:43.491833 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.491800    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7746-07025e"]
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492176    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerName="istio-proxy"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492187    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerName="istio-proxy"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492214    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d11737d-b322-4bef-9c56-5d1bcb52d5cd" containerName="payload-pre-processing"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492220    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d11737d-b322-4bef-9c56-5d1bcb52d5cd" containerName="payload-pre-processing"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492229    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a823a95-f61d-40c9-aaca-238000c3e77e" containerName="istio-proxy"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492235    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a823a95-f61d-40c9-aaca-238000c3e77e" containerName="istio-proxy"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492242    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a2b3f5c-e605-4049-829f-5ec2438677c7" containerName="cleanup"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492247    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a2b3f5c-e605-4049-829f-5ec2438677c7" containerName="cleanup"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492263    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4122da61-e79e-434b-83df-15053d1edeb5" containerName="istio-proxy"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492268    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4122da61-e79e-434b-83df-15053d1edeb5" containerName="istio-proxy"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492279    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a823a95-f61d-40c9-aaca-238000c3e77e" containerName="istio-proxy"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492284    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a823a95-f61d-40c9-aaca-238000c3e77e" containerName="istio-proxy"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492291    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b01fd2c-9199-4288-b509-029da1d8b9a6" containerName="payload-processing"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492297    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b01fd2c-9199-4288-b509-029da1d8b9a6" containerName="payload-processing"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492303    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d524126a-97d5-4888-ad42-d850ab79e8a5" containerName="maas-api"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492308    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d524126a-97d5-4888-ad42-d850ab79e8a5" containerName="maas-api"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492313    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4122da61-e79e-434b-83df-15053d1edeb5" containerName="istio-proxy"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492317    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4122da61-e79e-434b-83df-15053d1edeb5" containerName="istio-proxy"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492323    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3f2d1a-1fc8-4176-aaef-613d952479ad" containerName="cleanup"
Sep 17 03:47:43.492336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492328    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3f2d1a-1fc8-4176-aaef-613d952479ad" containerName="cleanup"
Sep 17 03:47:43.493158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492385    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerName="istio-proxy"
Sep 17 03:47:43.493158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492403    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d11737d-b322-4bef-9c56-5d1bcb52d5cd" containerName="payload-pre-processing"
Sep 17 03:47:43.493158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492410    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b01fd2c-9199-4288-b509-029da1d8b9a6" containerName="payload-processing"
Sep 17 03:47:43.493158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492417    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d524126a-97d5-4888-ad42-d850ab79e8a5" containerName="maas-api"
Sep 17 03:47:43.493158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492422    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4122da61-e79e-434b-83df-15053d1edeb5" containerName="istio-proxy"
Sep 17 03:47:43.493158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492428    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a823a95-f61d-40c9-aaca-238000c3e77e" containerName="istio-proxy"
Sep 17 03:47:43.493158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492434    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a3f2d1a-1fc8-4176-aaef-613d952479ad" containerName="cleanup"
Sep 17 03:47:43.493158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492439    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4122da61-e79e-434b-83df-15053d1edeb5" containerName="istio-proxy"
Sep 17 03:47:43.493158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492445    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a823a95-f61d-40c9-aaca-238000c3e77e" containerName="istio-proxy"
Sep 17 03:47:43.493158 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.492449    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a2b3f5c-e605-4049-829f-5ec2438677c7" containerName="cleanup"
Sep 17 03:47:43.495567 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.495548    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-07025e"
Sep 17 03:47:43.497882 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.497852    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s27lz\""
Sep 17 03:47:43.500010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.499986    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7746-07025e"]
Sep 17 03:47:43.608032 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.608010    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwcb9\" (UniqueName: \"kubernetes.io/projected/01e6e337-5c6b-4943-94b7-527f4f01e7b6-kube-api-access-vwcb9\") pod \"test-curl-7746-07025e\" (UID: \"01e6e337-5c6b-4943-94b7-527f4f01e7b6\") " pod="opendatahub/test-curl-7746-07025e"
Sep 17 03:47:43.708444 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.708422    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwcb9\" (UniqueName: \"kubernetes.io/projected/01e6e337-5c6b-4943-94b7-527f4f01e7b6-kube-api-access-vwcb9\") pod \"test-curl-7746-07025e\" (UID: \"01e6e337-5c6b-4943-94b7-527f4f01e7b6\") " pod="opendatahub/test-curl-7746-07025e"
Sep 17 03:47:43.715829 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.715801    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwcb9\" (UniqueName: \"kubernetes.io/projected/01e6e337-5c6b-4943-94b7-527f4f01e7b6-kube-api-access-vwcb9\") pod \"test-curl-7746-07025e\" (UID: \"01e6e337-5c6b-4943-94b7-527f4f01e7b6\") " pod="opendatahub/test-curl-7746-07025e"
Sep 17 03:47:43.806083 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:43.806031    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-07025e"
Sep 17 03:47:44.075110 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:44.075087    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7746-07025e"]
Sep 17 03:47:44.076992 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:47:44.076959    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01e6e337_5c6b_4943_94b7_527f4f01e7b6.slice/crio-ab657a6712ceb20d3aff021b958b8b1ad1a3a8d2c9ce9864dd16edea0194bcf5 WatchSource:0}: Error finding container ab657a6712ceb20d3aff021b958b8b1ad1a3a8d2c9ce9864dd16edea0194bcf5: Status 404 returned error can't find the container with id ab657a6712ceb20d3aff021b958b8b1ad1a3a8d2c9ce9864dd16edea0194bcf5
Sep 17 03:47:44.637665 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:44.637636    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7746-07025e" event={"ID":"01e6e337-5c6b-4943-94b7-527f4f01e7b6","Type":"ContainerStarted","Data":"ab657a6712ceb20d3aff021b958b8b1ad1a3a8d2c9ce9864dd16edea0194bcf5"}
Sep 17 03:47:46.651250 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:46.651215    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7746-07025e" event={"ID":"01e6e337-5c6b-4943-94b7-527f4f01e7b6","Type":"ContainerStarted","Data":"1a59b8cfa1f8ff12601d7159933bfab2d9b696cc2f13093cf16be80f9297f617"}
Sep 17 03:47:46.664507 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:46.664465    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7746-07025e" podStartSLOduration=1.583589331 podStartE2EDuration="3.664450116s" totalImagesPullingTime="2.080860785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:47:46.663896031 +0000 UTC m=+1717.266992036" watchObservedRunningTime="2026-09-17 03:47:46.664450116 +0000 UTC m=+1717.267546116"
Sep 17 03:47:47.243520 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.243495    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7746-07025e"]
Sep 17 03:47:47.248321 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.248297    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7746-07025e"]
Sep 17 03:47:47.359294 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.359232    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7746-07968c"]
Sep 17 03:47:47.359808 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.359792    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01e6e337-5c6b-4943-94b7-527f4f01e7b6" containerName="test-curl-7746-07025e"
Sep 17 03:47:47.359859 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.359812    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="01e6e337-5c6b-4943-94b7-527f4f01e7b6" containerName="test-curl-7746-07025e"
Sep 17 03:47:47.359859 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.359853    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerName="istio-proxy"
Sep 17 03:47:47.359925 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.359861    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerName="istio-proxy"
Sep 17 03:47:47.359959 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.359949    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="740bc52a-9b17-4fe7-ba69-fe88ab56706d" containerName="istio-proxy"
Sep 17 03:47:47.359993 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.359962    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="01e6e337-5c6b-4943-94b7-527f4f01e7b6" containerName="test-curl-7746-07025e"
Sep 17 03:47:47.363421 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.363404    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-07968c"
Sep 17 03:47:47.367056 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.367031    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7746-07968c"]
Sep 17 03:47:47.443070 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.443042    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k868s\" (UniqueName: \"kubernetes.io/projected/428029b2-5492-4b5a-a6d6-7b6176d24044-kube-api-access-k868s\") pod \"test-curl-7746-07968c\" (UID: \"428029b2-5492-4b5a-a6d6-7b6176d24044\") " pod="opendatahub/test-curl-7746-07968c"
Sep 17 03:47:47.544028 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.544003    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k868s\" (UniqueName: \"kubernetes.io/projected/428029b2-5492-4b5a-a6d6-7b6176d24044-kube-api-access-k868s\") pod \"test-curl-7746-07968c\" (UID: \"428029b2-5492-4b5a-a6d6-7b6176d24044\") " pod="opendatahub/test-curl-7746-07968c"
Sep 17 03:47:47.551472 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.551448    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k868s\" (UniqueName: \"kubernetes.io/projected/428029b2-5492-4b5a-a6d6-7b6176d24044-kube-api-access-k868s\") pod \"test-curl-7746-07968c\" (UID: \"428029b2-5492-4b5a-a6d6-7b6176d24044\") " pod="opendatahub/test-curl-7746-07968c"
Sep 17 03:47:47.657448 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:47:47.657425    2557 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7746-07025e" container="test-curl-7746-07025e"
Sep 17 03:47:47.674902 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.674884    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-07968c"
Sep 17 03:47:47.943448 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:47.943419    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7746-07968c"]
Sep 17 03:47:47.945404 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:47:47.945373    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod428029b2_5492_4b5a_a6d6_7b6176d24044.slice/crio-811943e680712261b8dc10668fc5094f596bf5827395830b3fdd05410ee183ff WatchSource:0}: Error finding container 811943e680712261b8dc10668fc5094f596bf5827395830b3fdd05410ee183ff: Status 404 returned error can't find the container with id 811943e680712261b8dc10668fc5094f596bf5827395830b3fdd05410ee183ff
Sep 17 03:47:48.663900 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:48.663864    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7746-07968c" event={"ID":"428029b2-5492-4b5a-a6d6-7b6176d24044","Type":"ContainerStarted","Data":"ffdd826e8a6d1d30fcdfc86110ca2284bb13659c9cfba089c27c5d164b589fcd"}
Sep 17 03:47:48.664325 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:48.663916    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7746-07968c" event={"ID":"428029b2-5492-4b5a-a6d6-7b6176d24044","Type":"ContainerStarted","Data":"811943e680712261b8dc10668fc5094f596bf5827395830b3fdd05410ee183ff"}
Sep 17 03:47:48.664325 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:48.663990    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7746-07025e" podUID="01e6e337-5c6b-4943-94b7-527f4f01e7b6" containerName="test-curl-7746-07025e" containerID="cri-o://1a59b8cfa1f8ff12601d7159933bfab2d9b696cc2f13093cf16be80f9297f617" gracePeriod=2
Sep 17 03:47:48.677227 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:48.677172    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7746-07968c" podStartSLOduration=1.677156084 podStartE2EDuration="1.677156084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:47:48.676392123 +0000 UTC m=+1719.279488130" watchObservedRunningTime="2026-09-17 03:47:48.677156084 +0000 UTC m=+1719.280252084"
Sep 17 03:47:49.268471 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:49.268441    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7746-07968c"]
Sep 17 03:47:49.273853 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:49.273827    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7746-07968c"]
Sep 17 03:47:49.670345 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:47:49.670317    2557 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7746-07968c" container="test-curl-7746-07968c"
Sep 17 03:47:50.675052 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:50.675017    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7746-07968c" podUID="428029b2-5492-4b5a-a6d6-7b6176d24044" containerName="test-curl-7746-07968c" containerID="cri-o://ffdd826e8a6d1d30fcdfc86110ca2284bb13659c9cfba089c27c5d164b589fcd" gracePeriod=2
Sep 17 03:47:50.849980 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:50.849957    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-07025e"
Sep 17 03:47:50.972949 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:50.972888    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwcb9\" (UniqueName: \"kubernetes.io/projected/01e6e337-5c6b-4943-94b7-527f4f01e7b6-kube-api-access-vwcb9\") pod \"01e6e337-5c6b-4943-94b7-527f4f01e7b6\" (UID: \"01e6e337-5c6b-4943-94b7-527f4f01e7b6\") "
Sep 17 03:47:50.974789 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:50.974764    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e6e337-5c6b-4943-94b7-527f4f01e7b6-kube-api-access-vwcb9" (OuterVolumeSpecName: "kube-api-access-vwcb9") pod "01e6e337-5c6b-4943-94b7-527f4f01e7b6" (UID: "01e6e337-5c6b-4943-94b7-527f4f01e7b6"). InnerVolumeSpecName "kube-api-access-vwcb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:47:51.074303 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:51.074278    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwcb9\" (UniqueName: \"kubernetes.io/projected/01e6e337-5c6b-4943-94b7-527f4f01e7b6-kube-api-access-vwcb9\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:47:51.680994 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:51.680963    2557 generic.go:356] "Generic (PLEG): container finished" podID="01e6e337-5c6b-4943-94b7-527f4f01e7b6" containerID="1a59b8cfa1f8ff12601d7159933bfab2d9b696cc2f13093cf16be80f9297f617" exitCode=137
Sep 17 03:47:51.681463 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:51.681003    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-07025e"
Sep 17 03:47:51.681463 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:51.681060    2557 scope.go:117] "RemoveContainer" containerID="1a59b8cfa1f8ff12601d7159933bfab2d9b696cc2f13093cf16be80f9297f617"
Sep 17 03:47:51.690398 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:51.690382    2557 scope.go:117] "RemoveContainer" containerID="1a59b8cfa1f8ff12601d7159933bfab2d9b696cc2f13093cf16be80f9297f617"
Sep 17 03:47:51.690648 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:47:51.690631    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a59b8cfa1f8ff12601d7159933bfab2d9b696cc2f13093cf16be80f9297f617\": container with ID starting with 1a59b8cfa1f8ff12601d7159933bfab2d9b696cc2f13093cf16be80f9297f617 not found: ID does not exist" containerID="1a59b8cfa1f8ff12601d7159933bfab2d9b696cc2f13093cf16be80f9297f617"
Sep 17 03:47:51.690695 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:51.690657    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a59b8cfa1f8ff12601d7159933bfab2d9b696cc2f13093cf16be80f9297f617"} err="failed to get container status \"1a59b8cfa1f8ff12601d7159933bfab2d9b696cc2f13093cf16be80f9297f617\": rpc error: code = NotFound desc = could not find container \"1a59b8cfa1f8ff12601d7159933bfab2d9b696cc2f13093cf16be80f9297f617\": container with ID starting with 1a59b8cfa1f8ff12601d7159933bfab2d9b696cc2f13093cf16be80f9297f617 not found: ID does not exist"
Sep 17 03:47:51.691229 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:47:51.691180    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7746-07025e\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="01e6e337-5c6b-4943-94b7-527f4f01e7b6" pod="opendatahub/test-curl-7746-07025e"
Sep 17 03:47:51.914540 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:51.914514    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01e6e337-5c6b-4943-94b7-527f4f01e7b6" path="/var/lib/kubelet/pods/01e6e337-5c6b-4943-94b7-527f4f01e7b6/volumes"
Sep 17 03:47:52.846479 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:52.846461    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-07968c"
Sep 17 03:47:52.991834 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:52.991778    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k868s\" (UniqueName: \"kubernetes.io/projected/428029b2-5492-4b5a-a6d6-7b6176d24044-kube-api-access-k868s\") pod \"428029b2-5492-4b5a-a6d6-7b6176d24044\" (UID: \"428029b2-5492-4b5a-a6d6-7b6176d24044\") "
Sep 17 03:47:52.993679 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:52.993654    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428029b2-5492-4b5a-a6d6-7b6176d24044-kube-api-access-k868s" (OuterVolumeSpecName: "kube-api-access-k868s") pod "428029b2-5492-4b5a-a6d6-7b6176d24044" (UID: "428029b2-5492-4b5a-a6d6-7b6176d24044"). InnerVolumeSpecName "kube-api-access-k868s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:47:53.092551 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:53.092526    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k868s\" (UniqueName: \"kubernetes.io/projected/428029b2-5492-4b5a-a6d6-7b6176d24044-kube-api-access-k868s\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:47:53.694533 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:53.694506    2557 generic.go:356] "Generic (PLEG): container finished" podID="428029b2-5492-4b5a-a6d6-7b6176d24044" containerID="ffdd826e8a6d1d30fcdfc86110ca2284bb13659c9cfba089c27c5d164b589fcd" exitCode=137
Sep 17 03:47:53.694640 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:53.694544    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-07968c"
Sep 17 03:47:53.694640 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:53.694607    2557 scope.go:117] "RemoveContainer" containerID="ffdd826e8a6d1d30fcdfc86110ca2284bb13659c9cfba089c27c5d164b589fcd"
Sep 17 03:47:53.703463 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:53.703448    2557 scope.go:117] "RemoveContainer" containerID="ffdd826e8a6d1d30fcdfc86110ca2284bb13659c9cfba089c27c5d164b589fcd"
Sep 17 03:47:53.703762 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:47:53.703738    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffdd826e8a6d1d30fcdfc86110ca2284bb13659c9cfba089c27c5d164b589fcd\": container with ID starting with ffdd826e8a6d1d30fcdfc86110ca2284bb13659c9cfba089c27c5d164b589fcd not found: ID does not exist" containerID="ffdd826e8a6d1d30fcdfc86110ca2284bb13659c9cfba089c27c5d164b589fcd"
Sep 17 03:47:53.703823 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:53.703774    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffdd826e8a6d1d30fcdfc86110ca2284bb13659c9cfba089c27c5d164b589fcd"} err="failed to get container status \"ffdd826e8a6d1d30fcdfc86110ca2284bb13659c9cfba089c27c5d164b589fcd\": rpc error: code = NotFound desc = could not find container \"ffdd826e8a6d1d30fcdfc86110ca2284bb13659c9cfba089c27c5d164b589fcd\": container with ID starting with ffdd826e8a6d1d30fcdfc86110ca2284bb13659c9cfba089c27c5d164b589fcd not found: ID does not exist"
Sep 17 03:47:53.703981 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:47:53.703964    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7746-07968c\" is forbidden: User \"system:node:ip-10-0-130-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-153.ec2.internal' and this object" podUID="428029b2-5492-4b5a-a6d6-7b6176d24044" pod="opendatahub/test-curl-7746-07968c"
Sep 17 03:47:53.914274 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:53.914253    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="428029b2-5492-4b5a-a6d6-7b6176d24044" path="/var/lib/kubelet/pods/428029b2-5492-4b5a-a6d6-7b6176d24044/volumes"
Sep 17 03:47:55.185907 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.185873    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"]
Sep 17 03:47:55.186394 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.186334    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="428029b2-5492-4b5a-a6d6-7b6176d24044" containerName="test-curl-7746-07968c"
Sep 17 03:47:55.186394 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.186348    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="428029b2-5492-4b5a-a6d6-7b6176d24044" containerName="test-curl-7746-07968c"
Sep 17 03:47:55.186521 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.186434    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="428029b2-5492-4b5a-a6d6-7b6176d24044" containerName="test-curl-7746-07968c"
Sep 17 03:47:55.190870 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.190850    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.193104 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.193082    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-22e045\"/\"openshift-service-ca.crt\""
Sep 17 03:47:55.193219 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.193140    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-22e045\"/\"rate-test-model-22e045-kserve-self-signed-certs\""
Sep 17 03:47:55.193857 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.193835    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-22e045\"/\"kube-root-ca.crt\""
Sep 17 03:47:55.193965 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.193885    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-22e045\"/\"default-dockercfg-8dd44\""
Sep 17 03:47:55.200013 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.199991    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"]
Sep 17 03:47:55.307224 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.307173    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-model-cache\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.307336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.307231    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07add368-1e38-42d3-9253-de6e46737b16-tls-certs\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.307336 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.307262    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-dshm\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.307431 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.307404    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-home\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.307468 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.307447    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-tmp-dir\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.307506 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.307480    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2v7j\" (UniqueName: \"kubernetes.io/projected/07add368-1e38-42d3-9253-de6e46737b16-kube-api-access-f2v7j\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.408882 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.408847    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-home\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.408882 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.408878    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-tmp-dir\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.409086 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.408900    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2v7j\" (UniqueName: \"kubernetes.io/projected/07add368-1e38-42d3-9253-de6e46737b16-kube-api-access-f2v7j\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.409086 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.408985    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-model-cache\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.409086 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.409012    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07add368-1e38-42d3-9253-de6e46737b16-tls-certs\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.409086 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.409034    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-dshm\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.409338 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.409275    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-tmp-dir\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.409338 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.409314    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-home\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.409445 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.409394    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-model-cache\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.411187 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.411157    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-dshm\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.411532 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.411512    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07add368-1e38-42d3-9253-de6e46737b16-tls-certs\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.417503 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.417484    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2v7j\" (UniqueName: \"kubernetes.io/projected/07add368-1e38-42d3-9253-de6e46737b16-kube-api-access-f2v7j\") pod \"rate-test-model-22e045-kserve-86bfb58c79-wpsss\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") " pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.501515 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.501454    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:47:55.773519 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:55.773493    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"]
Sep 17 03:47:55.775241 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:47:55.775206    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07add368_1e38_42d3_9253_de6e46737b16.slice/crio-d294b4a126f3355f85d232f58e82fd805c77d533b1264600bb8baf9f08cea640 WatchSource:0}: Error finding container d294b4a126f3355f85d232f58e82fd805c77d533b1264600bb8baf9f08cea640: Status 404 returned error can't find the container with id d294b4a126f3355f85d232f58e82fd805c77d533b1264600bb8baf9f08cea640
Sep 17 03:47:56.713786 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:56.713750    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss" event={"ID":"07add368-1e38-42d3-9253-de6e46737b16","Type":"ContainerStarted","Data":"9e5c0b4341ee1d3f0ad51403d9dba1308dee0c6be1732319533635d75e7b5d7e"}
Sep 17 03:47:56.714149 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:56.713799    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss" event={"ID":"07add368-1e38-42d3-9253-de6e46737b16","Type":"ContainerStarted","Data":"d294b4a126f3355f85d232f58e82fd805c77d533b1264600bb8baf9f08cea640"}
Sep 17 03:47:56.730885 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:47:56.730844    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss" podStartSLOduration=1.730829597 podStartE2EDuration="1.730829597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:47:56.728986304 +0000 UTC m=+1727.332082309" watchObservedRunningTime="2026-09-17 03:47:56.730829597 +0000 UTC m=+1727.333925601"
Sep 17 03:48:05.501577 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:48:05.501541    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:48:05.501577 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:48:05.501583    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:48:05.513613 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:48:05.513591    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:48:05.773822 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:48:05.773759    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:49:05.661162 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:05.661076    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"]
Sep 17 03:49:07.262494 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.262457    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"]
Sep 17 03:49:07.266801 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.266781    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.268883 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.268858    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d0ab1a-openshift-default-dockercfg-rrshx\""
Sep 17 03:49:07.278510 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.278486    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"]
Sep 17 03:49:07.301040 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.301016    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-workload-certs\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.301135 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.301056    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-istio-data\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.301135 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.301087    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-credential-socket\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.301135 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.301107    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmj8x\" (UniqueName: \"kubernetes.io/projected/d634cb58-e29d-4bb5-9135-de8381a94411-kube-api-access-qmj8x\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.301135 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.301133    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d634cb58-e29d-4bb5-9135-de8381a94411-istiod-ca-cert\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.301391 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.301281    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d634cb58-e29d-4bb5-9135-de8381a94411-istio-podinfo\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.301391 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.301316    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-workload-socket\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.301498 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.301393    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-istio-envoy\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.301498 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.301417    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d634cb58-e29d-4bb5-9135-de8381a94411-istio-token\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.402553 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.402507    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d634cb58-e29d-4bb5-9135-de8381a94411-istio-podinfo\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.402553 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.402555    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-workload-socket\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.402780 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.402608    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-istio-envoy\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.402780 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.402631    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d634cb58-e29d-4bb5-9135-de8381a94411-istio-token\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.402780 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.402678    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-workload-certs\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.402780 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.402717    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-istio-data\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.402780 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.402758    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-credential-socket\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.403134 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.402786    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmj8x\" (UniqueName: \"kubernetes.io/projected/d634cb58-e29d-4bb5-9135-de8381a94411-kube-api-access-qmj8x\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.403134 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.402835    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d634cb58-e29d-4bb5-9135-de8381a94411-istiod-ca-cert\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.403134 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.402967    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-workload-socket\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.403134 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.403117    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-workload-certs\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.403407 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.403363    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-credential-socket\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.403407 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.403385    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-istio-data\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.403598 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.403577    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d634cb58-e29d-4bb5-9135-de8381a94411-istiod-ca-cert\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.405272 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.405250    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d634cb58-e29d-4bb5-9135-de8381a94411-istio-podinfo\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.405442 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.405424    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-istio-envoy\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.410139 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.410121    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d634cb58-e29d-4bb5-9135-de8381a94411-istio-token\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.410267 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.410171    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmj8x\" (UniqueName: \"kubernetes.io/projected/d634cb58-e29d-4bb5-9135-de8381a94411-kube-api-access-qmj8x\") pod \"e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") " pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.580370 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.580303    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:07.857966 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.857939    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"]
Sep 17 03:49:07.859116 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:49:07.859094    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd634cb58_e29d_4bb5_9135_de8381a94411.slice/crio-1a00d58ac6de74217beb2b381aff3176b7b8104abfca00a8d19aa157cc22b104 WatchSource:0}: Error finding container 1a00d58ac6de74217beb2b381aff3176b7b8104abfca00a8d19aa157cc22b104: Status 404 returned error can't find the container with id 1a00d58ac6de74217beb2b381aff3176b7b8104abfca00a8d19aa157cc22b104
Sep 17 03:49:07.861452 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.861415    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:49:07.861540 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.861485    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:49:07.861540 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:07.861511    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:49:08.110620 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:08.110549    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p" event={"ID":"d634cb58-e29d-4bb5-9135-de8381a94411","Type":"ContainerStarted","Data":"35c1e3ac77000b560b7d19f44b171b6c15633198c433ec154958237d933f1922"}
Sep 17 03:49:08.110620 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:08.110615    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p" event={"ID":"d634cb58-e29d-4bb5-9135-de8381a94411","Type":"ContainerStarted","Data":"1a00d58ac6de74217beb2b381aff3176b7b8104abfca00a8d19aa157cc22b104"}
Sep 17 03:49:08.129160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:08.129119    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p" podStartSLOduration=1.129100662 podStartE2EDuration="1.129100662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:49:08.127154401 +0000 UTC m=+1798.730250408" watchObservedRunningTime="2026-09-17 03:49:08.129100662 +0000 UTC m=+1798.732196665"
Sep 17 03:49:08.580508 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:08.580479    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:08.585660 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:08.585635    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:09.116881 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:09.116853    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:09.117910 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:09.117883    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:10.027151 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:10.027040    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:49:10.033058 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:10.033031    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:49:11.970155 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:11.970123    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"]
Sep 17 03:49:11.973902 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:11.973882    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"
Sep 17 03:49:11.975926 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:11.975909    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d0ab1a\""
Sep 17 03:49:11.982322 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:11.982297    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"]
Sep 17 03:49:12.046976 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:12.046949    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a280dc9-dffc-4b55-a27e-4d8a093a8cd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn\" (UID: \"2a280dc9-dffc-4b55-a27e-4d8a093a8cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"
Sep 17 03:49:12.047098 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:12.046981    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g7pm\" (UniqueName: \"kubernetes.io/projected/2a280dc9-dffc-4b55-a27e-4d8a093a8cd1-kube-api-access-7g7pm\") pod \"payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn\" (UID: \"2a280dc9-dffc-4b55-a27e-4d8a093a8cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"
Sep 17 03:49:12.148248 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:12.148189    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a280dc9-dffc-4b55-a27e-4d8a093a8cd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn\" (UID: \"2a280dc9-dffc-4b55-a27e-4d8a093a8cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"
Sep 17 03:49:12.148248 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:12.148254    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g7pm\" (UniqueName: \"kubernetes.io/projected/2a280dc9-dffc-4b55-a27e-4d8a093a8cd1-kube-api-access-7g7pm\") pod \"payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn\" (UID: \"2a280dc9-dffc-4b55-a27e-4d8a093a8cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"
Sep 17 03:49:12.148942 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:12.148918    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a280dc9-dffc-4b55-a27e-4d8a093a8cd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn\" (UID: \"2a280dc9-dffc-4b55-a27e-4d8a093a8cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"
Sep 17 03:49:12.156075 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:12.156056    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g7pm\" (UniqueName: \"kubernetes.io/projected/2a280dc9-dffc-4b55-a27e-4d8a093a8cd1-kube-api-access-7g7pm\") pod \"payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn\" (UID: \"2a280dc9-dffc-4b55-a27e-4d8a093a8cd1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"
Sep 17 03:49:12.286355 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:12.286296    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"
Sep 17 03:49:12.556890 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:12.556860    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"]
Sep 17 03:49:12.558185 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:49:12.558159    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a280dc9_dffc_4b55_a27e_4d8a093a8cd1.slice/crio-dbbf4d4d418b9b5690d613ce0bf978d6a95edef226bc8686bd71629b53485777 WatchSource:0}: Error finding container dbbf4d4d418b9b5690d613ce0bf978d6a95edef226bc8686bd71629b53485777: Status 404 returned error can't find the container with id dbbf4d4d418b9b5690d613ce0bf978d6a95edef226bc8686bd71629b53485777
Sep 17 03:49:13.139565 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:13.139530    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn" event={"ID":"2a280dc9-dffc-4b55-a27e-4d8a093a8cd1","Type":"ContainerStarted","Data":"5c8d85bd6c9d4a08c3165ae0434f6b45dd45a3ce7ca17a86820a7c1279c05a72"}
Sep 17 03:49:13.140019 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:13.139594    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn" event={"ID":"2a280dc9-dffc-4b55-a27e-4d8a093a8cd1","Type":"ContainerStarted","Data":"dbbf4d4d418b9b5690d613ce0bf978d6a95edef226bc8686bd71629b53485777"}
Sep 17 03:49:13.140019 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:13.139625    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"
Sep 17 03:49:13.157143 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:13.157089    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn" podStartSLOduration=2.1570698520000002 podStartE2EDuration="2.157069852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:49:13.156263462 +0000 UTC m=+1803.759359474" watchObservedRunningTime="2026-09-17 03:49:13.157069852 +0000 UTC m=+1803.760165861"
Sep 17 03:49:20.681146 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:20.681100    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss" podUID="07add368-1e38-42d3-9253-de6e46737b16" containerName="main" containerID="cri-o://9e5c0b4341ee1d3f0ad51403d9dba1308dee0c6be1732319533635d75e7b5d7e" gracePeriod=45
Sep 17 03:49:21.003663 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.003638    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:49:21.152693 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.152659    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-model-cache\") pod \"07add368-1e38-42d3-9253-de6e46737b16\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") "
Sep 17 03:49:21.152885 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.152723    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-tmp-dir\") pod \"07add368-1e38-42d3-9253-de6e46737b16\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") "
Sep 17 03:49:21.152885 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.152775    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07add368-1e38-42d3-9253-de6e46737b16-tls-certs\") pod \"07add368-1e38-42d3-9253-de6e46737b16\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") "
Sep 17 03:49:21.152885 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.152816    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2v7j\" (UniqueName: \"kubernetes.io/projected/07add368-1e38-42d3-9253-de6e46737b16-kube-api-access-f2v7j\") pod \"07add368-1e38-42d3-9253-de6e46737b16\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") "
Sep 17 03:49:21.152885 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.152863    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-dshm\") pod \"07add368-1e38-42d3-9253-de6e46737b16\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") "
Sep 17 03:49:21.153116 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.152898    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-model-cache" (OuterVolumeSpecName: "model-cache") pod "07add368-1e38-42d3-9253-de6e46737b16" (UID: "07add368-1e38-42d3-9253-de6e46737b16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:49:21.153116 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.152937    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-home\") pod \"07add368-1e38-42d3-9253-de6e46737b16\" (UID: \"07add368-1e38-42d3-9253-de6e46737b16\") "
Sep 17 03:49:21.153116 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.152957    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07add368-1e38-42d3-9253-de6e46737b16" (UID: "07add368-1e38-42d3-9253-de6e46737b16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:49:21.153311 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.153236    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-home" (OuterVolumeSpecName: "home") pod "07add368-1e38-42d3-9253-de6e46737b16" (UID: "07add368-1e38-42d3-9253-de6e46737b16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:49:21.153778 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.153755    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-home\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:49:21.153778 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.153779    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-model-cache\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:49:21.153961 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.153793    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-tmp-dir\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:49:21.155318 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.155284    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07add368-1e38-42d3-9253-de6e46737b16-kube-api-access-f2v7j" (OuterVolumeSpecName: "kube-api-access-f2v7j") pod "07add368-1e38-42d3-9253-de6e46737b16" (UID: "07add368-1e38-42d3-9253-de6e46737b16"). InnerVolumeSpecName "kube-api-access-f2v7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:49:21.155448 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.155388    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-dshm" (OuterVolumeSpecName: "dshm") pod "07add368-1e38-42d3-9253-de6e46737b16" (UID: "07add368-1e38-42d3-9253-de6e46737b16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:49:21.155448 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.155391    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07add368-1e38-42d3-9253-de6e46737b16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07add368-1e38-42d3-9253-de6e46737b16" (UID: "07add368-1e38-42d3-9253-de6e46737b16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:49:21.185420 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.185393    2557 generic.go:356] "Generic (PLEG): container finished" podID="07add368-1e38-42d3-9253-de6e46737b16" containerID="9e5c0b4341ee1d3f0ad51403d9dba1308dee0c6be1732319533635d75e7b5d7e" exitCode=0
Sep 17 03:49:21.185511 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.185460    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"
Sep 17 03:49:21.185511 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.185474    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss" event={"ID":"07add368-1e38-42d3-9253-de6e46737b16","Type":"ContainerDied","Data":"9e5c0b4341ee1d3f0ad51403d9dba1308dee0c6be1732319533635d75e7b5d7e"}
Sep 17 03:49:21.185610 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.185519    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss" event={"ID":"07add368-1e38-42d3-9253-de6e46737b16","Type":"ContainerDied","Data":"d294b4a126f3355f85d232f58e82fd805c77d533b1264600bb8baf9f08cea640"}
Sep 17 03:49:21.185610 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.185540    2557 scope.go:117] "RemoveContainer" containerID="9e5c0b4341ee1d3f0ad51403d9dba1308dee0c6be1732319533635d75e7b5d7e"
Sep 17 03:49:21.194248 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.194230    2557 scope.go:117] "RemoveContainer" containerID="9e5c0b4341ee1d3f0ad51403d9dba1308dee0c6be1732319533635d75e7b5d7e"
Sep 17 03:49:21.194519 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:49:21.194500    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5c0b4341ee1d3f0ad51403d9dba1308dee0c6be1732319533635d75e7b5d7e\": container with ID starting with 9e5c0b4341ee1d3f0ad51403d9dba1308dee0c6be1732319533635d75e7b5d7e not found: ID does not exist" containerID="9e5c0b4341ee1d3f0ad51403d9dba1308dee0c6be1732319533635d75e7b5d7e"
Sep 17 03:49:21.194594 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.194531    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5c0b4341ee1d3f0ad51403d9dba1308dee0c6be1732319533635d75e7b5d7e"} err="failed to get container status \"9e5c0b4341ee1d3f0ad51403d9dba1308dee0c6be1732319533635d75e7b5d7e\": rpc error: code = NotFound desc = could not find container \"9e5c0b4341ee1d3f0ad51403d9dba1308dee0c6be1732319533635d75e7b5d7e\": container with ID starting with 9e5c0b4341ee1d3f0ad51403d9dba1308dee0c6be1732319533635d75e7b5d7e not found: ID does not exist"
Sep 17 03:49:21.208521 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.208498    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"]
Sep 17 03:49:21.211919 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.211898    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-22e045/rate-test-model-22e045-kserve-86bfb58c79-wpsss"]
Sep 17 03:49:21.254841 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.254781    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07add368-1e38-42d3-9253-de6e46737b16-tls-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:49:21.254841 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.254803    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2v7j\" (UniqueName: \"kubernetes.io/projected/07add368-1e38-42d3-9253-de6e46737b16-kube-api-access-f2v7j\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:49:21.254841 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.254813    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07add368-1e38-42d3-9253-de6e46737b16-dshm\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:49:21.917555 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:21.917482    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07add368-1e38-42d3-9253-de6e46737b16" path="/var/lib/kubelet/pods/07add368-1e38-42d3-9253-de6e46737b16/volumes"
Sep 17 03:49:24.147364 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:24.147330    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"
Sep 17 03:49:24.486482 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:24.485690    2557 patch_prober.go:28] interesting pod/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": read tcp 10.132.0.2:59948->10.132.0.89:15021: read: connection reset by peer" start-of-body=
Sep 17 03:49:24.486482 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:24.485750    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p" podUID="d634cb58-e29d-4bb5-9135-de8381a94411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": read tcp 10.132.0.2:59948->10.132.0.89:15021: read: connection reset by peer"
Sep 17 03:49:25.212391 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:25.212362    2557 generic.go:356] "Generic (PLEG): container finished" podID="d634cb58-e29d-4bb5-9135-de8381a94411" containerID="35c1e3ac77000b560b7d19f44b171b6c15633198c433ec154958237d933f1922" exitCode=0
Sep 17 03:49:25.212760 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:25.212439    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p" event={"ID":"d634cb58-e29d-4bb5-9135-de8381a94411","Type":"ContainerDied","Data":"35c1e3ac77000b560b7d19f44b171b6c15633198c433ec154958237d933f1922"}
Sep 17 03:49:25.212850 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:25.212831    2557 scope.go:117] "RemoveContainer" containerID="35c1e3ac77000b560b7d19f44b171b6c15633198c433ec154958237d933f1922"
Sep 17 03:49:25.213908 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:25.213871    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:49:25.214027 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:25.213938    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:49:25.214027 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:25.213972    2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 17 03:49:26.220268 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:26.220225    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p" event={"ID":"d634cb58-e29d-4bb5-9135-de8381a94411","Type":"ContainerStarted","Data":"e396af62dae2d80e4fb50c0e2ead85e335fbd80a268a464ada137e1449ff977f"}
Sep 17 03:49:26.581010 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:26.580925    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:26.581162 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:26.581078    2557 patch_prober.go:28] interesting pod/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Sep 17 03:49:26.581162 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:26.581130    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p" podUID="d634cb58-e29d-4bb5-9135-de8381a94411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Sep 17 03:49:27.581012 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:27.580978    2557 patch_prober.go:28] interesting pod/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Sep 17 03:49:27.581420 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:27.581034    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p" podUID="d634cb58-e29d-4bb5-9135-de8381a94411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Sep 17 03:49:28.581640 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:28.581605    2557 patch_prober.go:28] interesting pod/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Sep 17 03:49:28.582027 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:28.581659    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p" podUID="d634cb58-e29d-4bb5-9135-de8381a94411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Sep 17 03:49:29.580929 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:29.580895    2557 patch_prober.go:28] interesting pod/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Sep 17 03:49:29.581110 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:29.580951    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p" podUID="d634cb58-e29d-4bb5-9135-de8381a94411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Sep 17 03:49:30.580878 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:30.580842    2557 patch_prober.go:28] interesting pod/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Sep 17 03:49:30.581270 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:30.580898    2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p" podUID="d634cb58-e29d-4bb5-9135-de8381a94411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Sep 17 03:49:31.584266 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:31.584234    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:31.584731 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:31.584486    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:31.585129 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:31.585114    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:49:40.897868 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:40.897835    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"]
Sep 17 03:49:40.899174 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:40.898543    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07add368-1e38-42d3-9253-de6e46737b16" containerName="main"
Sep 17 03:49:40.899174 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:40.898570    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="07add368-1e38-42d3-9253-de6e46737b16" containerName="main"
Sep 17 03:49:40.899394 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:40.899224    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="07add368-1e38-42d3-9253-de6e46737b16" containerName="main"
Sep 17 03:49:40.903635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:40.903613    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:40.905781 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:40.905737    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-84cd5d-serving-cert\""
Sep 17 03:49:40.913337 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:40.913283    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"]
Sep 17 03:49:41.036271 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.036234    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e7548a-c4f8-440a-9497-741ae9640730-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:41.036271 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.036269    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c8rp\" (UniqueName: \"kubernetes.io/projected/52e7548a-c4f8-440a-9497-741ae9640730-kube-api-access-5c8rp\") pod \"maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:41.036532 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.036296    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e7548a-c4f8-440a-9497-741ae9640730-metrics-certs\") pod \"maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:41.036532 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.036511    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e7548a-c4f8-440a-9497-741ae9640730-maas-api-tls\") pod \"maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:41.137400 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.137369    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e7548a-c4f8-440a-9497-741ae9640730-metrics-certs\") pod \"maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:41.137557 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.137438    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e7548a-c4f8-440a-9497-741ae9640730-maas-api-tls\") pod \"maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:41.137557 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.137540    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e7548a-c4f8-440a-9497-741ae9640730-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:41.137686 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.137565    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c8rp\" (UniqueName: \"kubernetes.io/projected/52e7548a-c4f8-440a-9497-741ae9640730-kube-api-access-5c8rp\") pod \"maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:41.138142 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.138117    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e7548a-c4f8-440a-9497-741ae9640730-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:41.139969 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.139943    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e7548a-c4f8-440a-9497-741ae9640730-maas-api-tls\") pod \"maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:41.140060 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.139943    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e7548a-c4f8-440a-9497-741ae9640730-metrics-certs\") pod \"maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:41.144838 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.144818    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c8rp\" (UniqueName: \"kubernetes.io/projected/52e7548a-c4f8-440a-9497-741ae9640730-kube-api-access-5c8rp\") pod \"maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:41.217711 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.217658    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:41.495850 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.495821    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"]
Sep 17 03:49:41.496795 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:49:41.496767    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e7548a_c4f8_440a_9497_741ae9640730.slice/crio-8620bcecabec8d23c85c3a2b1ea943e3c51d881fe214d39488453b06307641c4 WatchSource:0}: Error finding container 8620bcecabec8d23c85c3a2b1ea943e3c51d881fe214d39488453b06307641c4: Status 404 returned error can't find the container with id 8620bcecabec8d23c85c3a2b1ea943e3c51d881fe214d39488453b06307641c4
Sep 17 03:49:41.498327 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.498307    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 03:49:41.524508 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.524481    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"]
Sep 17 03:49:41.529244 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.529228    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"
Sep 17 03:49:41.531242 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.531219    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-84cd5d-dockercfg-wrkx6\""
Sep 17 03:49:41.531355 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.531269    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-84cd5d\""
Sep 17 03:49:41.540225 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.540204    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"]
Sep 17 03:49:41.641622 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.641581    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a831fc84-6101-417c-b3a0-ad97ea3df6e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64\" (UID: \"a831fc84-6101-417c-b3a0-ad97ea3df6e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"
Sep 17 03:49:41.641796 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.641649    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62jqh\" (UniqueName: \"kubernetes.io/projected/a831fc84-6101-417c-b3a0-ad97ea3df6e4-kube-api-access-62jqh\") pod \"payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64\" (UID: \"a831fc84-6101-417c-b3a0-ad97ea3df6e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"
Sep 17 03:49:41.742917 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.742885    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a831fc84-6101-417c-b3a0-ad97ea3df6e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64\" (UID: \"a831fc84-6101-417c-b3a0-ad97ea3df6e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"
Sep 17 03:49:41.743098 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.742934    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62jqh\" (UniqueName: \"kubernetes.io/projected/a831fc84-6101-417c-b3a0-ad97ea3df6e4-kube-api-access-62jqh\") pod \"payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64\" (UID: \"a831fc84-6101-417c-b3a0-ad97ea3df6e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"
Sep 17 03:49:41.743699 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.743639    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a831fc84-6101-417c-b3a0-ad97ea3df6e4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64\" (UID: \"a831fc84-6101-417c-b3a0-ad97ea3df6e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"
Sep 17 03:49:41.751588 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.751568    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62jqh\" (UniqueName: \"kubernetes.io/projected/a831fc84-6101-417c-b3a0-ad97ea3df6e4-kube-api-access-62jqh\") pod \"payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64\" (UID: \"a831fc84-6101-417c-b3a0-ad97ea3df6e4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"
Sep 17 03:49:41.840956 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:41.840925    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"
Sep 17 03:49:42.115067 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:42.115041    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"]
Sep 17 03:49:42.116583 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:49:42.116556    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda831fc84_6101_417c_b3a0_ad97ea3df6e4.slice/crio-e66271594101416a60338656ee2fa42975700943ce28d7f739f94d03ff40fab4 WatchSource:0}: Error finding container e66271594101416a60338656ee2fa42975700943ce28d7f739f94d03ff40fab4: Status 404 returned error can't find the container with id e66271594101416a60338656ee2fa42975700943ce28d7f739f94d03ff40fab4
Sep 17 03:49:42.321845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:42.321808    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2" event={"ID":"52e7548a-c4f8-440a-9497-741ae9640730","Type":"ContainerStarted","Data":"8f7f53904ede1eb5178ae605ac40dc3c438019fd8f4d2bd75127433bcfe39749"}
Sep 17 03:49:42.322022 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:42.321875    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2" event={"ID":"52e7548a-c4f8-440a-9497-741ae9640730","Type":"ContainerStarted","Data":"8620bcecabec8d23c85c3a2b1ea943e3c51d881fe214d39488453b06307641c4"}
Sep 17 03:49:42.322022 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:42.321905    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:42.323800 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:42.323774    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64" event={"ID":"a831fc84-6101-417c-b3a0-ad97ea3df6e4","Type":"ContainerStarted","Data":"b372be653e87fc01e4b05adab8db17be8d9672cadcdd39298883cd4b691563cb"}
Sep 17 03:49:42.323925 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:42.323827    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"
Sep 17 03:49:42.323925 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:42.323851    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64" event={"ID":"a831fc84-6101-417c-b3a0-ad97ea3df6e4","Type":"ContainerStarted","Data":"e66271594101416a60338656ee2fa42975700943ce28d7f739f94d03ff40fab4"}
Sep 17 03:49:42.337989 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:42.337939    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2" podStartSLOduration=2.042891116 podStartE2EDuration="2.337925317s" totalImagesPullingTime="295.034201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:49:42.33573263 +0000 UTC m=+1832.938828657" watchObservedRunningTime="2026-09-17 03:49:42.337925317 +0000 UTC m=+1832.941021321"
Sep 17 03:49:42.350717 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:42.350671    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64" podStartSLOduration=1.35065343 podStartE2EDuration="1.35065343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:49:42.350161116 +0000 UTC m=+1832.953257117" watchObservedRunningTime="2026-09-17 03:49:42.35065343 +0000 UTC m=+1832.953749434"
Sep 17 03:49:48.334720 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:48.334690    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:49:53.331484 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:49:53.331451    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"
Sep 17 03:50:08.898590 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:08.898548    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7gscq_6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb/payload-processing/0.log"
Sep 17 03:50:09.031889 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:09.031863    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64_a831fc84-6101-417c-b3a0-ad97ea3df6e4/payload-processing/0.log"
Sep 17 03:50:10.295499 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:10.295471    2557 scope.go:117] "RemoveContainer" containerID="dd1bd18e9a13a62a73581855e6cd2c5bab4b10a67934a308b525d8487684b255"
Sep 17 03:50:10.303729 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:10.303707    2557 scope.go:117] "RemoveContainer" containerID="3d5de7a61bf19ef532155351087d70a5ff5793a1ef9e99a646ee3a741cb06e07"
Sep 17 03:50:50.283765 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:50.283736    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7gscq_6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb/payload-processing/0.log"
Sep 17 03:50:50.408508 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:50.408477    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64_a831fc84-6101-417c-b3a0-ad97ea3df6e4/payload-processing/0.log"
Sep 17 03:50:50.543121 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:50.543056    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64_a831fc84-6101-417c-b3a0-ad97ea3df6e4/payload-processing/0.log"
Sep 17 03:50:57.389329 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:57.389292    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"]
Sep 17 03:50:57.394019 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:57.393990    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:50:57.396015 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:57.395983    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-0ccfd0-serving-cert\""
Sep 17 03:50:57.404739 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:57.404714    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"]
Sep 17 03:50:57.486724 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:57.486697    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcx8s\" (UniqueName: \"kubernetes.io/projected/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-kube-api-access-lcx8s\") pod \"maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:50:57.486840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:57.486737    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:50:57.486840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:57.486774    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:50:57.486840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:57.486802    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:50:57.588247 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:57.588222    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcx8s\" (UniqueName: \"kubernetes.io/projected/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-kube-api-access-lcx8s\") pod \"maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:50:57.588394 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:57.588264    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:50:57.588394 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:57.588289    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:50:57.588394 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:57.588310    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:50:57.588512 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:50:57.588463    2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-serving-cert: secret "maas-api-e2e-ipp-cleanup-0ccfd0-serving-cert" not found
Sep 17 03:50:57.588553 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:50:57.588547    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-maas-api-tls podName:c4e8d375-d954-4c6b-bb58-d88afd16cdd5 nodeName:}" failed. No retries permitted until 2026-09-17 03:50:58.08852495 +0000 UTC m=+1908.691620945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k" (UID: "c4e8d375-d954-4c6b-bb58-d88afd16cdd5") : secret "maas-api-e2e-ipp-cleanup-0ccfd0-serving-cert" not found
Sep 17 03:50:57.589084 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:57.589066    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:50:57.591006 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:57.590983    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:50:57.595840 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:57.595819    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcx8s\" (UniqueName: \"kubernetes.io/projected/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-kube-api-access-lcx8s\") pod \"maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:50:58.094146 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:58.094114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:50:58.096449 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:58.096417    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:50:58.307530 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:58.307498    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:50:58.581221 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:58.581166    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"]
Sep 17 03:50:58.582558 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:50:58.582530    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e8d375_d954_4c6b_bb58_d88afd16cdd5.slice/crio-c25dd37b219c7f137148b81f39bc4527055bb2c43573a24ad119d2498eb861dc WatchSource:0}: Error finding container c25dd37b219c7f137148b81f39bc4527055bb2c43573a24ad119d2498eb861dc: Status 404 returned error can't find the container with id c25dd37b219c7f137148b81f39bc4527055bb2c43573a24ad119d2498eb861dc
Sep 17 03:50:58.755011 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:58.754983    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k" event={"ID":"c4e8d375-d954-4c6b-bb58-d88afd16cdd5","Type":"ContainerStarted","Data":"c25dd37b219c7f137148b81f39bc4527055bb2c43573a24ad119d2498eb861dc"}
Sep 17 03:50:59.761726 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:59.761683    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k" event={"ID":"c4e8d375-d954-4c6b-bb58-d88afd16cdd5","Type":"ContainerStarted","Data":"92f4c34883050ae668275a17e9667d98a81718fad7f3eefa4eb6a92e20e2e926"}
Sep 17 03:50:59.762112 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:50:59.761758    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:51:05.772695 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:05.772664    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:51:05.788241 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:05.787992    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k" podStartSLOduration=8.448405802 podStartE2EDuration="8.787973666s" totalImagesPullingTime="339.567864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:50:59.776899155 +0000 UTC m=+1910.379995161" watchObservedRunningTime="2026-09-17 03:51:05.787973666 +0000 UTC m=+1916.391069672"
Sep 17 03:51:10.331254 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:10.331213    2557 scope.go:117] "RemoveContainer" containerID="ca4b5433c32bbf4474f8845662878ec2ed943167a9697bd610d2a70582fabc4a"
Sep 17 03:51:10.339519 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:10.339500    2557 scope.go:117] "RemoveContainer" containerID="e9a7104fe6711bdc61210e09d973b496a69d219ccd36ffc70dd9a7b1d49c385e"
Sep 17 03:51:10.347374 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:10.347351    2557 scope.go:117] "RemoveContainer" containerID="39d017df5a9f935bf7c4968a2437f5f3d2c9da0a91270b5ed7976219ab75dcb3"
Sep 17 03:51:25.971652 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:25.971618    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"]
Sep 17 03:51:25.972285 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:25.971859    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k" podUID="c4e8d375-d954-4c6b-bb58-d88afd16cdd5" containerName="maas-api" containerID="cri-o://92f4c34883050ae668275a17e9667d98a81718fad7f3eefa4eb6a92e20e2e926" gracePeriod=30
Sep 17 03:51:26.225498 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.225443    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:51:26.242103 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.242078    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-openshift-service-ca\") pod \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") "
Sep 17 03:51:26.242278 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.242152    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-metrics-certs\") pod \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") "
Sep 17 03:51:26.242415 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.242313    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-maas-api-tls\") pod \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") "
Sep 17 03:51:26.242485 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.242415    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcx8s\" (UniqueName: \"kubernetes.io/projected/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-kube-api-access-lcx8s\") pod \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\" (UID: \"c4e8d375-d954-4c6b-bb58-d88afd16cdd5\") "
Sep 17 03:51:26.243601 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.243570    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4e8d375-d954-4c6b-bb58-d88afd16cdd5" (UID: "c4e8d375-d954-4c6b-bb58-d88afd16cdd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:51:26.245131 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.245086    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4e8d375-d954-4c6b-bb58-d88afd16cdd5" (UID: "c4e8d375-d954-4c6b-bb58-d88afd16cdd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:51:26.245332 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.245305    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c4e8d375-d954-4c6b-bb58-d88afd16cdd5" (UID: "c4e8d375-d954-4c6b-bb58-d88afd16cdd5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:51:26.245436 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.245359    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-kube-api-access-lcx8s" (OuterVolumeSpecName: "kube-api-access-lcx8s") pod "c4e8d375-d954-4c6b-bb58-d88afd16cdd5" (UID: "c4e8d375-d954-4c6b-bb58-d88afd16cdd5"). InnerVolumeSpecName "kube-api-access-lcx8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:51:26.344447 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.344407    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-openshift-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:51:26.344447 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.344442    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-metrics-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:51:26.344447 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.344453    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-maas-api-tls\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:51:26.344735 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.344461    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcx8s\" (UniqueName: \"kubernetes.io/projected/c4e8d375-d954-4c6b-bb58-d88afd16cdd5-kube-api-access-lcx8s\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:51:26.914081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.914047    2557 generic.go:356] "Generic (PLEG): container finished" podID="c4e8d375-d954-4c6b-bb58-d88afd16cdd5" containerID="92f4c34883050ae668275a17e9667d98a81718fad7f3eefa4eb6a92e20e2e926" exitCode=0
Sep 17 03:51:26.914081 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.914078    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"
Sep 17 03:51:26.914322 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.914098    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k" event={"ID":"c4e8d375-d954-4c6b-bb58-d88afd16cdd5","Type":"ContainerDied","Data":"92f4c34883050ae668275a17e9667d98a81718fad7f3eefa4eb6a92e20e2e926"}
Sep 17 03:51:26.914322 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.914141    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k" event={"ID":"c4e8d375-d954-4c6b-bb58-d88afd16cdd5","Type":"ContainerDied","Data":"c25dd37b219c7f137148b81f39bc4527055bb2c43573a24ad119d2498eb861dc"}
Sep 17 03:51:26.914322 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.914178    2557 scope.go:117] "RemoveContainer" containerID="92f4c34883050ae668275a17e9667d98a81718fad7f3eefa4eb6a92e20e2e926"
Sep 17 03:51:26.924275 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.924243    2557 scope.go:117] "RemoveContainer" containerID="92f4c34883050ae668275a17e9667d98a81718fad7f3eefa4eb6a92e20e2e926"
Sep 17 03:51:26.924560 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:51:26.924537    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f4c34883050ae668275a17e9667d98a81718fad7f3eefa4eb6a92e20e2e926\": container with ID starting with 92f4c34883050ae668275a17e9667d98a81718fad7f3eefa4eb6a92e20e2e926 not found: ID does not exist" containerID="92f4c34883050ae668275a17e9667d98a81718fad7f3eefa4eb6a92e20e2e926"
Sep 17 03:51:26.924645 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.924568    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f4c34883050ae668275a17e9667d98a81718fad7f3eefa4eb6a92e20e2e926"} err="failed to get container status \"92f4c34883050ae668275a17e9667d98a81718fad7f3eefa4eb6a92e20e2e926\": rpc error: code = NotFound desc = could not find container \"92f4c34883050ae668275a17e9667d98a81718fad7f3eefa4eb6a92e20e2e926\": container with ID starting with 92f4c34883050ae668275a17e9667d98a81718fad7f3eefa4eb6a92e20e2e926 not found: ID does not exist"
Sep 17 03:51:26.935524 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.935500    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"]
Sep 17 03:51:26.939157 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:26.939137    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0ccfd0-5b6cb55cdf-2cb7k"]
Sep 17 03:51:27.914704 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:27.914668    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4e8d375-d954-4c6b-bb58-d88afd16cdd5" path="/var/lib/kubelet/pods/c4e8d375-d954-4c6b-bb58-d88afd16cdd5/volumes"
Sep 17 03:51:52.068523 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:52.068488    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"]
Sep 17 03:51:52.068952 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:52.068730    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn" podUID="2a280dc9-dffc-4b55-a27e-4d8a093a8cd1" containerName="payload-pre-processing" containerID="cri-o://5c8d85bd6c9d4a08c3165ae0434f6b45dd45a3ce7ca17a86820a7c1279c05a72" gracePeriod=30
Sep 17 03:51:52.348140 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:52.348103    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"
Sep 17 03:51:52.483717 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:52.483682    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a280dc9-dffc-4b55-a27e-4d8a093a8cd1-plugins-config-volume\") pod \"2a280dc9-dffc-4b55-a27e-4d8a093a8cd1\" (UID: \"2a280dc9-dffc-4b55-a27e-4d8a093a8cd1\") "
Sep 17 03:51:52.483879 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:52.483767    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g7pm\" (UniqueName: \"kubernetes.io/projected/2a280dc9-dffc-4b55-a27e-4d8a093a8cd1-kube-api-access-7g7pm\") pod \"2a280dc9-dffc-4b55-a27e-4d8a093a8cd1\" (UID: \"2a280dc9-dffc-4b55-a27e-4d8a093a8cd1\") "
Sep 17 03:51:52.484024 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:52.484002    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a280dc9-dffc-4b55-a27e-4d8a093a8cd1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a280dc9-dffc-4b55-a27e-4d8a093a8cd1" (UID: "2a280dc9-dffc-4b55-a27e-4d8a093a8cd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:51:52.485815 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:52.485784    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a280dc9-dffc-4b55-a27e-4d8a093a8cd1-kube-api-access-7g7pm" (OuterVolumeSpecName: "kube-api-access-7g7pm") pod "2a280dc9-dffc-4b55-a27e-4d8a093a8cd1" (UID: "2a280dc9-dffc-4b55-a27e-4d8a093a8cd1"). InnerVolumeSpecName "kube-api-access-7g7pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:51:52.584721 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:52.584659    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a280dc9-dffc-4b55-a27e-4d8a093a8cd1-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:51:52.584721 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:52.584680    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g7pm\" (UniqueName: \"kubernetes.io/projected/2a280dc9-dffc-4b55-a27e-4d8a093a8cd1-kube-api-access-7g7pm\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:51:53.058261 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:53.058229    2557 generic.go:356] "Generic (PLEG): container finished" podID="2a280dc9-dffc-4b55-a27e-4d8a093a8cd1" containerID="5c8d85bd6c9d4a08c3165ae0434f6b45dd45a3ce7ca17a86820a7c1279c05a72" exitCode=0
Sep 17 03:51:53.058261 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:53.058254    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn" event={"ID":"2a280dc9-dffc-4b55-a27e-4d8a093a8cd1","Type":"ContainerDied","Data":"5c8d85bd6c9d4a08c3165ae0434f6b45dd45a3ce7ca17a86820a7c1279c05a72"}
Sep 17 03:51:53.058473 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:53.058304    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn" event={"ID":"2a280dc9-dffc-4b55-a27e-4d8a093a8cd1","Type":"ContainerDied","Data":"dbbf4d4d418b9b5690d613ce0bf978d6a95edef226bc8686bd71629b53485777"}
Sep 17 03:51:53.058473 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:53.058239    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"
Sep 17 03:51:53.058473 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:53.058331    2557 scope.go:117] "RemoveContainer" containerID="5c8d85bd6c9d4a08c3165ae0434f6b45dd45a3ce7ca17a86820a7c1279c05a72"
Sep 17 03:51:53.068911 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:53.068888    2557 scope.go:117] "RemoveContainer" containerID="5c8d85bd6c9d4a08c3165ae0434f6b45dd45a3ce7ca17a86820a7c1279c05a72"
Sep 17 03:51:53.069252 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:51:53.069181    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8d85bd6c9d4a08c3165ae0434f6b45dd45a3ce7ca17a86820a7c1279c05a72\": container with ID starting with 5c8d85bd6c9d4a08c3165ae0434f6b45dd45a3ce7ca17a86820a7c1279c05a72 not found: ID does not exist" containerID="5c8d85bd6c9d4a08c3165ae0434f6b45dd45a3ce7ca17a86820a7c1279c05a72"
Sep 17 03:51:53.069521 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:53.069250    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8d85bd6c9d4a08c3165ae0434f6b45dd45a3ce7ca17a86820a7c1279c05a72"} err="failed to get container status \"5c8d85bd6c9d4a08c3165ae0434f6b45dd45a3ce7ca17a86820a7c1279c05a72\": rpc error: code = NotFound desc = could not find container \"5c8d85bd6c9d4a08c3165ae0434f6b45dd45a3ce7ca17a86820a7c1279c05a72\": container with ID starting with 5c8d85bd6c9d4a08c3165ae0434f6b45dd45a3ce7ca17a86820a7c1279c05a72 not found: ID does not exist"
Sep 17 03:51:53.081738 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:53.081712    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"]
Sep 17 03:51:53.085281 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:53.085262    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d0ab1a-6494cfb974-g7bmn"]
Sep 17 03:51:53.917095 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:51:53.917057    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a280dc9-dffc-4b55-a27e-4d8a093a8cd1" path="/var/lib/kubelet/pods/2a280dc9-dffc-4b55-a27e-4d8a093a8cd1/volumes"
Sep 17 03:53:03.697438 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:03.697403    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"]
Sep 17 03:53:03.697984 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:03.697624    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p" podUID="d634cb58-e29d-4bb5-9135-de8381a94411" containerName="istio-proxy" containerID="cri-o://e396af62dae2d80e4fb50c0e2ead85e335fbd80a268a464ada137e1449ff977f" gracePeriod=30
Sep 17 03:53:08.953971 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:08.953944    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:53:09.121236 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.121210    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d634cb58-e29d-4bb5-9135-de8381a94411-istio-token\") pod \"d634cb58-e29d-4bb5-9135-de8381a94411\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") "
Sep 17 03:53:09.121376 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.121355    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-istio-data\") pod \"d634cb58-e29d-4bb5-9135-de8381a94411\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") "
Sep 17 03:53:09.121439 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.121387    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d634cb58-e29d-4bb5-9135-de8381a94411-istiod-ca-cert\") pod \"d634cb58-e29d-4bb5-9135-de8381a94411\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") "
Sep 17 03:53:09.121501 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.121451    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-workload-socket\") pod \"d634cb58-e29d-4bb5-9135-de8381a94411\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") "
Sep 17 03:53:09.121501 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.121480    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-credential-socket\") pod \"d634cb58-e29d-4bb5-9135-de8381a94411\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") "
Sep 17 03:53:09.121696 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.121505    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmj8x\" (UniqueName: \"kubernetes.io/projected/d634cb58-e29d-4bb5-9135-de8381a94411-kube-api-access-qmj8x\") pod \"d634cb58-e29d-4bb5-9135-de8381a94411\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") "
Sep 17 03:53:09.121696 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.121574    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-workload-certs\") pod \"d634cb58-e29d-4bb5-9135-de8381a94411\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") "
Sep 17 03:53:09.121822 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.121745    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d634cb58-e29d-4bb5-9135-de8381a94411-istio-podinfo\") pod \"d634cb58-e29d-4bb5-9135-de8381a94411\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") "
Sep 17 03:53:09.121822 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.121751    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d634cb58-e29d-4bb5-9135-de8381a94411" (UID: "d634cb58-e29d-4bb5-9135-de8381a94411"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:53:09.121822 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.121779    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d634cb58-e29d-4bb5-9135-de8381a94411" (UID: "d634cb58-e29d-4bb5-9135-de8381a94411"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:53:09.121822 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.121805    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-istio-envoy\") pod \"d634cb58-e29d-4bb5-9135-de8381a94411\" (UID: \"d634cb58-e29d-4bb5-9135-de8381a94411\") "
Sep 17 03:53:09.122023 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.121845    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d634cb58-e29d-4bb5-9135-de8381a94411-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d634cb58-e29d-4bb5-9135-de8381a94411" (UID: "d634cb58-e29d-4bb5-9135-de8381a94411"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:53:09.122023 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.121876    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d634cb58-e29d-4bb5-9135-de8381a94411" (UID: "d634cb58-e29d-4bb5-9135-de8381a94411"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:53:09.122160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.122135    2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d634cb58-e29d-4bb5-9135-de8381a94411-istiod-ca-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:09.122160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.121666    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-istio-data" (OuterVolumeSpecName: "istio-data") pod "d634cb58-e29d-4bb5-9135-de8381a94411" (UID: "d634cb58-e29d-4bb5-9135-de8381a94411"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:53:09.122160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.122163    2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-workload-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:09.122435 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.122178    2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-credential-socket\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:09.122435 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.122203    2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-workload-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:09.123492 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.123470    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d634cb58-e29d-4bb5-9135-de8381a94411-istio-token" (OuterVolumeSpecName: "istio-token") pod "d634cb58-e29d-4bb5-9135-de8381a94411" (UID: "d634cb58-e29d-4bb5-9135-de8381a94411"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:53:09.123603 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.123584    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d634cb58-e29d-4bb5-9135-de8381a94411-kube-api-access-qmj8x" (OuterVolumeSpecName: "kube-api-access-qmj8x") pod "d634cb58-e29d-4bb5-9135-de8381a94411" (UID: "d634cb58-e29d-4bb5-9135-de8381a94411"). InnerVolumeSpecName "kube-api-access-qmj8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:53:09.123706 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.123690    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d634cb58-e29d-4bb5-9135-de8381a94411-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d634cb58-e29d-4bb5-9135-de8381a94411" (UID: "d634cb58-e29d-4bb5-9135-de8381a94411"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 03:53:09.123895 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.123877    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d634cb58-e29d-4bb5-9135-de8381a94411" (UID: "d634cb58-e29d-4bb5-9135-de8381a94411"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 03:53:09.223560 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.223537    2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d634cb58-e29d-4bb5-9135-de8381a94411-istio-token\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:09.223560 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.223557    2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-istio-data\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:09.223698 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.223566    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmj8x\" (UniqueName: \"kubernetes.io/projected/d634cb58-e29d-4bb5-9135-de8381a94411-kube-api-access-qmj8x\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:09.223698 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.223575    2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d634cb58-e29d-4bb5-9135-de8381a94411-istio-podinfo\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:09.223698 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.223583    2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d634cb58-e29d-4bb5-9135-de8381a94411-istio-envoy\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:09.456135 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.456076    2557 generic.go:356] "Generic (PLEG): container finished" podID="d634cb58-e29d-4bb5-9135-de8381a94411" containerID="e396af62dae2d80e4fb50c0e2ead85e335fbd80a268a464ada137e1449ff977f" exitCode=0
Sep 17 03:53:09.456135 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.456135    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"
Sep 17 03:53:09.456302 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.456172    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p" event={"ID":"d634cb58-e29d-4bb5-9135-de8381a94411","Type":"ContainerDied","Data":"e396af62dae2d80e4fb50c0e2ead85e335fbd80a268a464ada137e1449ff977f"}
Sep 17 03:53:09.456302 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.456246    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p" event={"ID":"d634cb58-e29d-4bb5-9135-de8381a94411","Type":"ContainerDied","Data":"1a00d58ac6de74217beb2b381aff3176b7b8104abfca00a8d19aa157cc22b104"}
Sep 17 03:53:09.456302 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.456273    2557 scope.go:117] "RemoveContainer" containerID="e396af62dae2d80e4fb50c0e2ead85e335fbd80a268a464ada137e1449ff977f"
Sep 17 03:53:09.465703 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.465685    2557 scope.go:117] "RemoveContainer" containerID="35c1e3ac77000b560b7d19f44b171b6c15633198c433ec154958237d933f1922"
Sep 17 03:53:09.473724 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.473710    2557 scope.go:117] "RemoveContainer" containerID="e396af62dae2d80e4fb50c0e2ead85e335fbd80a268a464ada137e1449ff977f"
Sep 17 03:53:09.473988 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:53:09.473971    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e396af62dae2d80e4fb50c0e2ead85e335fbd80a268a464ada137e1449ff977f\": container with ID starting with e396af62dae2d80e4fb50c0e2ead85e335fbd80a268a464ada137e1449ff977f not found: ID does not exist" containerID="e396af62dae2d80e4fb50c0e2ead85e335fbd80a268a464ada137e1449ff977f"
Sep 17 03:53:09.474030 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.473995    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e396af62dae2d80e4fb50c0e2ead85e335fbd80a268a464ada137e1449ff977f"} err="failed to get container status \"e396af62dae2d80e4fb50c0e2ead85e335fbd80a268a464ada137e1449ff977f\": rpc error: code = NotFound desc = could not find container \"e396af62dae2d80e4fb50c0e2ead85e335fbd80a268a464ada137e1449ff977f\": container with ID starting with e396af62dae2d80e4fb50c0e2ead85e335fbd80a268a464ada137e1449ff977f not found: ID does not exist"
Sep 17 03:53:09.474030 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.474012    2557 scope.go:117] "RemoveContainer" containerID="35c1e3ac77000b560b7d19f44b171b6c15633198c433ec154958237d933f1922"
Sep 17 03:53:09.474256 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:53:09.474236    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35c1e3ac77000b560b7d19f44b171b6c15633198c433ec154958237d933f1922\": container with ID starting with 35c1e3ac77000b560b7d19f44b171b6c15633198c433ec154958237d933f1922 not found: ID does not exist" containerID="35c1e3ac77000b560b7d19f44b171b6c15633198c433ec154958237d933f1922"
Sep 17 03:53:09.474325 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.474265    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35c1e3ac77000b560b7d19f44b171b6c15633198c433ec154958237d933f1922"} err="failed to get container status \"35c1e3ac77000b560b7d19f44b171b6c15633198c433ec154958237d933f1922\": rpc error: code = NotFound desc = could not find container \"35c1e3ac77000b560b7d19f44b171b6c15633198c433ec154958237d933f1922\": container with ID starting with 35c1e3ac77000b560b7d19f44b171b6c15633198c433ec154958237d933f1922 not found: ID does not exist"
Sep 17 03:53:09.478367 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.478343    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"]
Sep 17 03:53:09.481726 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.481703    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d0ab1a-openshift-default-69d96d4cdd-h695p"]
Sep 17 03:53:09.915532 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:09.915511    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d634cb58-e29d-4bb5-9135-de8381a94411" path="/var/lib/kubelet/pods/d634cb58-e29d-4bb5-9135-de8381a94411/volumes"
Sep 17 03:53:14.454930 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.454899    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"]
Sep 17 03:53:14.455424 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.455168    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2" podUID="52e7548a-c4f8-440a-9497-741ae9640730" containerName="maas-api" containerID="cri-o://8f7f53904ede1eb5178ae605ac40dc3c438019fd8f4d2bd75127433bcfe39749" gracePeriod=30
Sep 17 03:53:14.523933 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.523902    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"]
Sep 17 03:53:14.524418 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.524382    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64" podUID="a831fc84-6101-417c-b3a0-ad97ea3df6e4" containerName="payload-processing" containerID="cri-o://b372be653e87fc01e4b05adab8db17be8d9672cadcdd39298883cd4b691563cb" gracePeriod=30
Sep 17 03:53:14.784117 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.784092    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:53:14.814698 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.814671    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"
Sep 17 03:53:14.867664 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.867638    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e7548a-c4f8-440a-9497-741ae9640730-metrics-certs\") pod \"52e7548a-c4f8-440a-9497-741ae9640730\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") "
Sep 17 03:53:14.867831 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.867680    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a831fc84-6101-417c-b3a0-ad97ea3df6e4-plugins-config-volume\") pod \"a831fc84-6101-417c-b3a0-ad97ea3df6e4\" (UID: \"a831fc84-6101-417c-b3a0-ad97ea3df6e4\") "
Sep 17 03:53:14.867831 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.867731    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c8rp\" (UniqueName: \"kubernetes.io/projected/52e7548a-c4f8-440a-9497-741ae9640730-kube-api-access-5c8rp\") pod \"52e7548a-c4f8-440a-9497-741ae9640730\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") "
Sep 17 03:53:14.867831 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.867762    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e7548a-c4f8-440a-9497-741ae9640730-openshift-service-ca\") pod \"52e7548a-c4f8-440a-9497-741ae9640730\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") "
Sep 17 03:53:14.867831 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.867787    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e7548a-c4f8-440a-9497-741ae9640730-maas-api-tls\") pod \"52e7548a-c4f8-440a-9497-741ae9640730\" (UID: \"52e7548a-c4f8-440a-9497-741ae9640730\") "
Sep 17 03:53:14.867831 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.867824    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62jqh\" (UniqueName: \"kubernetes.io/projected/a831fc84-6101-417c-b3a0-ad97ea3df6e4-kube-api-access-62jqh\") pod \"a831fc84-6101-417c-b3a0-ad97ea3df6e4\" (UID: \"a831fc84-6101-417c-b3a0-ad97ea3df6e4\") "
Sep 17 03:53:14.868249 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.868188    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a831fc84-6101-417c-b3a0-ad97ea3df6e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a831fc84-6101-417c-b3a0-ad97ea3df6e4" (UID: "a831fc84-6101-417c-b3a0-ad97ea3df6e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:53:14.868451 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.868418    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e7548a-c4f8-440a-9497-741ae9640730-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52e7548a-c4f8-440a-9497-741ae9640730" (UID: "52e7548a-c4f8-440a-9497-741ae9640730"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:53:14.869996 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.869971    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e7548a-c4f8-440a-9497-741ae9640730-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52e7548a-c4f8-440a-9497-741ae9640730" (UID: "52e7548a-c4f8-440a-9497-741ae9640730"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:53:14.870353 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.870333    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e7548a-c4f8-440a-9497-741ae9640730-kube-api-access-5c8rp" (OuterVolumeSpecName: "kube-api-access-5c8rp") pod "52e7548a-c4f8-440a-9497-741ae9640730" (UID: "52e7548a-c4f8-440a-9497-741ae9640730"). InnerVolumeSpecName "kube-api-access-5c8rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:53:14.870420 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.870335    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a831fc84-6101-417c-b3a0-ad97ea3df6e4-kube-api-access-62jqh" (OuterVolumeSpecName: "kube-api-access-62jqh") pod "a831fc84-6101-417c-b3a0-ad97ea3df6e4" (UID: "a831fc84-6101-417c-b3a0-ad97ea3df6e4"). InnerVolumeSpecName "kube-api-access-62jqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:53:14.870420 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.870338    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e7548a-c4f8-440a-9497-741ae9640730-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "52e7548a-c4f8-440a-9497-741ae9640730" (UID: "52e7548a-c4f8-440a-9497-741ae9640730"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:53:14.969089 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.969017    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e7548a-c4f8-440a-9497-741ae9640730-metrics-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:14.969089 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.969043    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a831fc84-6101-417c-b3a0-ad97ea3df6e4-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:14.969089 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.969057    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c8rp\" (UniqueName: \"kubernetes.io/projected/52e7548a-c4f8-440a-9497-741ae9640730-kube-api-access-5c8rp\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:14.969089 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.969069    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e7548a-c4f8-440a-9497-741ae9640730-openshift-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:14.969089 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.969077    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e7548a-c4f8-440a-9497-741ae9640730-maas-api-tls\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:14.969089 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:14.969086    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62jqh\" (UniqueName: \"kubernetes.io/projected/a831fc84-6101-417c-b3a0-ad97ea3df6e4-kube-api-access-62jqh\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:15.490518 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.490493    2557 generic.go:356] "Generic (PLEG): container finished" podID="a831fc84-6101-417c-b3a0-ad97ea3df6e4" containerID="b372be653e87fc01e4b05adab8db17be8d9672cadcdd39298883cd4b691563cb" exitCode=0
Sep 17 03:53:15.490905 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.490538    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"
Sep 17 03:53:15.490905 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.490590    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64" event={"ID":"a831fc84-6101-417c-b3a0-ad97ea3df6e4","Type":"ContainerDied","Data":"b372be653e87fc01e4b05adab8db17be8d9672cadcdd39298883cd4b691563cb"}
Sep 17 03:53:15.490905 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.490638    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64" event={"ID":"a831fc84-6101-417c-b3a0-ad97ea3df6e4","Type":"ContainerDied","Data":"e66271594101416a60338656ee2fa42975700943ce28d7f739f94d03ff40fab4"}
Sep 17 03:53:15.490905 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.490663    2557 scope.go:117] "RemoveContainer" containerID="b372be653e87fc01e4b05adab8db17be8d9672cadcdd39298883cd4b691563cb"
Sep 17 03:53:15.492299 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.492278    2557 generic.go:356] "Generic (PLEG): container finished" podID="52e7548a-c4f8-440a-9497-741ae9640730" containerID="8f7f53904ede1eb5178ae605ac40dc3c438019fd8f4d2bd75127433bcfe39749" exitCode=0
Sep 17 03:53:15.492408 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.492313    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2" event={"ID":"52e7548a-c4f8-440a-9497-741ae9640730","Type":"ContainerDied","Data":"8f7f53904ede1eb5178ae605ac40dc3c438019fd8f4d2bd75127433bcfe39749"}
Sep 17 03:53:15.492408 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.492329    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"
Sep 17 03:53:15.492408 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.492351    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2" event={"ID":"52e7548a-c4f8-440a-9497-741ae9640730","Type":"ContainerDied","Data":"8620bcecabec8d23c85c3a2b1ea943e3c51d881fe214d39488453b06307641c4"}
Sep 17 03:53:15.499357 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.499338    2557 scope.go:117] "RemoveContainer" containerID="b372be653e87fc01e4b05adab8db17be8d9672cadcdd39298883cd4b691563cb"
Sep 17 03:53:15.499648 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:53:15.499626    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b372be653e87fc01e4b05adab8db17be8d9672cadcdd39298883cd4b691563cb\": container with ID starting with b372be653e87fc01e4b05adab8db17be8d9672cadcdd39298883cd4b691563cb not found: ID does not exist" containerID="b372be653e87fc01e4b05adab8db17be8d9672cadcdd39298883cd4b691563cb"
Sep 17 03:53:15.499740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.499651    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b372be653e87fc01e4b05adab8db17be8d9672cadcdd39298883cd4b691563cb"} err="failed to get container status \"b372be653e87fc01e4b05adab8db17be8d9672cadcdd39298883cd4b691563cb\": rpc error: code = NotFound desc = could not find container \"b372be653e87fc01e4b05adab8db17be8d9672cadcdd39298883cd4b691563cb\": container with ID starting with b372be653e87fc01e4b05adab8db17be8d9672cadcdd39298883cd4b691563cb not found: ID does not exist"
Sep 17 03:53:15.499740 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.499667    2557 scope.go:117] "RemoveContainer" containerID="8f7f53904ede1eb5178ae605ac40dc3c438019fd8f4d2bd75127433bcfe39749"
Sep 17 03:53:15.508177 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.508162    2557 scope.go:117] "RemoveContainer" containerID="8f7f53904ede1eb5178ae605ac40dc3c438019fd8f4d2bd75127433bcfe39749"
Sep 17 03:53:15.508456 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:53:15.508437    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7f53904ede1eb5178ae605ac40dc3c438019fd8f4d2bd75127433bcfe39749\": container with ID starting with 8f7f53904ede1eb5178ae605ac40dc3c438019fd8f4d2bd75127433bcfe39749 not found: ID does not exist" containerID="8f7f53904ede1eb5178ae605ac40dc3c438019fd8f4d2bd75127433bcfe39749"
Sep 17 03:53:15.508516 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.508462    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7f53904ede1eb5178ae605ac40dc3c438019fd8f4d2bd75127433bcfe39749"} err="failed to get container status \"8f7f53904ede1eb5178ae605ac40dc3c438019fd8f4d2bd75127433bcfe39749\": rpc error: code = NotFound desc = could not find container \"8f7f53904ede1eb5178ae605ac40dc3c438019fd8f4d2bd75127433bcfe39749\": container with ID starting with 8f7f53904ede1eb5178ae605ac40dc3c438019fd8f4d2bd75127433bcfe39749 not found: ID does not exist"
Sep 17 03:53:15.515967 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.515947    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"]
Sep 17 03:53:15.523160 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.523138    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-84cd5d-85cdc5bb7f-ngf64"]
Sep 17 03:53:15.542936 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.542906    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"]
Sep 17 03:53:15.547630 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.547610    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-84cd5d-66444568cc-pgfq2"]
Sep 17 03:53:15.915471 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.915434    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52e7548a-c4f8-440a-9497-741ae9640730" path="/var/lib/kubelet/pods/52e7548a-c4f8-440a-9497-741ae9640730/volumes"
Sep 17 03:53:15.915845 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:15.915830    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a831fc84-6101-417c-b3a0-ad97ea3df6e4" path="/var/lib/kubelet/pods/a831fc84-6101-417c-b3a0-ad97ea3df6e4/volumes"
Sep 17 03:53:31.917714 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:31.917675    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-22e045-29826945-wnsd6"]
Sep 17 03:53:31.918178 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:31.917981    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-22e045-29826945-wnsd6"]
Sep 17 03:53:33.915094 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:33.915058    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a3f2d1a-1fc8-4176-aaef-613d952479ad" path="/var/lib/kubelet/pods/2a3f2d1a-1fc8-4176-aaef-613d952479ad/volumes"
Sep 17 03:53:49.449635 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.449600    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"]
Sep 17 03:53:49.450124 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.449840    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg" podUID="8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a" containerName="payload-processing" containerID="cri-o://b43e6299c18d035d966e09b887ec644d116c33b76678b8f4b7cd19342250a5d6" gracePeriod=30
Sep 17 03:53:49.458535 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.458506    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"]
Sep 17 03:53:49.458713 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.458694    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx" podUID="3e0dd334-39e0-4176-b826-fe45f297879f" containerName="payload-pre-processing" containerID="cri-o://c525152048cd5b8b5c474b3796514e0eaafed5baae211499a076fae18eee0815" gracePeriod=30
Sep 17 03:53:49.675140 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.675109    2557 generic.go:356] "Generic (PLEG): container finished" podID="3e0dd334-39e0-4176-b826-fe45f297879f" containerID="c525152048cd5b8b5c474b3796514e0eaafed5baae211499a076fae18eee0815" exitCode=0
Sep 17 03:53:49.675301 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.675162    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx" event={"ID":"3e0dd334-39e0-4176-b826-fe45f297879f","Type":"ContainerDied","Data":"c525152048cd5b8b5c474b3796514e0eaafed5baae211499a076fae18eee0815"}
Sep 17 03:53:49.689490 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.689439    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg" event={"ID":"8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a","Type":"ContainerDied","Data":"b43e6299c18d035d966e09b887ec644d116c33b76678b8f4b7cd19342250a5d6"}
Sep 17 03:53:49.689632 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.689458    2557 generic.go:356] "Generic (PLEG): container finished" podID="8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a" containerID="b43e6299c18d035d966e09b887ec644d116c33b76678b8f4b7cd19342250a5d6" exitCode=0
Sep 17 03:53:49.747459 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.747431    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"
Sep 17 03:53:49.749422 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.749402    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"
Sep 17 03:53:49.860179 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.860144    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a-plugins-config-volume\") pod \"8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a\" (UID: \"8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a\") "
Sep 17 03:53:49.860383 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.860227    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cqcc\" (UniqueName: \"kubernetes.io/projected/8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a-kube-api-access-9cqcc\") pod \"8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a\" (UID: \"8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a\") "
Sep 17 03:53:49.860383 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.860283    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0dd334-39e0-4176-b826-fe45f297879f-plugins-config-volume\") pod \"3e0dd334-39e0-4176-b826-fe45f297879f\" (UID: \"3e0dd334-39e0-4176-b826-fe45f297879f\") "
Sep 17 03:53:49.860383 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.860311    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7tvr\" (UniqueName: \"kubernetes.io/projected/3e0dd334-39e0-4176-b826-fe45f297879f-kube-api-access-h7tvr\") pod \"3e0dd334-39e0-4176-b826-fe45f297879f\" (UID: \"3e0dd334-39e0-4176-b826-fe45f297879f\") "
Sep 17 03:53:49.860583 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.860545    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a" (UID: "8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:53:49.860683 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.860656    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0dd334-39e0-4176-b826-fe45f297879f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e0dd334-39e0-4176-b826-fe45f297879f" (UID: "3e0dd334-39e0-4176-b826-fe45f297879f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:53:49.860767 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.860747    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:49.862473 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.862444    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0dd334-39e0-4176-b826-fe45f297879f-kube-api-access-h7tvr" (OuterVolumeSpecName: "kube-api-access-h7tvr") pod "3e0dd334-39e0-4176-b826-fe45f297879f" (UID: "3e0dd334-39e0-4176-b826-fe45f297879f"). InnerVolumeSpecName "kube-api-access-h7tvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:53:49.862683 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.862666    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a-kube-api-access-9cqcc" (OuterVolumeSpecName: "kube-api-access-9cqcc") pod "8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a" (UID: "8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a"). InnerVolumeSpecName "kube-api-access-9cqcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:53:49.961512 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.961472    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cqcc\" (UniqueName: \"kubernetes.io/projected/8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a-kube-api-access-9cqcc\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:49.961512 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.961504    2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e0dd334-39e0-4176-b826-fe45f297879f-plugins-config-volume\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:49.961512 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:49.961517    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7tvr\" (UniqueName: \"kubernetes.io/projected/3e0dd334-39e0-4176-b826-fe45f297879f-kube-api-access-h7tvr\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:50.695034 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:50.695004    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"
Sep 17 03:53:50.695034 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:50.695029    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg" event={"ID":"8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a","Type":"ContainerDied","Data":"3cef1230bfb8673cb21c2afe3b34580fbc27c4a739fc4ce34bdc560de1f9ddcd"}
Sep 17 03:53:50.695516 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:50.695092    2557 scope.go:117] "RemoveContainer" containerID="b43e6299c18d035d966e09b887ec644d116c33b76678b8f4b7cd19342250a5d6"
Sep 17 03:53:50.696486 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:50.696462    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"
Sep 17 03:53:50.696562 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:50.696498    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx" event={"ID":"3e0dd334-39e0-4176-b826-fe45f297879f","Type":"ContainerDied","Data":"8d978a2fd55f388d1ab663de55f91b25c36fb7d160ea71626434366dd6cc1dfd"}
Sep 17 03:53:50.703989 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:50.703975    2557 scope.go:117] "RemoveContainer" containerID="c525152048cd5b8b5c474b3796514e0eaafed5baae211499a076fae18eee0815"
Sep 17 03:53:50.713478 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:50.713458    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"]
Sep 17 03:53:50.717820 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:50.717797    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f6d581-86d6dd978b-n88jx"]
Sep 17 03:53:50.726361 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:50.726315    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"]
Sep 17 03:53:50.730274 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:50.730253    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f6d581-59d4b5485c-xpvxg"]
Sep 17 03:53:51.914802 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:51.914770    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e0dd334-39e0-4176-b826-fe45f297879f" path="/var/lib/kubelet/pods/3e0dd334-39e0-4176-b826-fe45f297879f/volumes"
Sep 17 03:53:51.915237 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:51.915131    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a" path="/var/lib/kubelet/pods/8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a/volumes"
Sep 17 03:53:59.057334 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.057302    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c8c8d6b99-xc8jh"]
Sep 17 03:53:59.057718 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.057478    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh" podUID="b9c51d77-1a0c-41df-aa7a-43c46772e496" containerName="manager" containerID="cri-o://b64d21b3110e6e66fea9dacb8f14d429f9b4a54848935b967fe6748ffe5911ac" gracePeriod=30
Sep 17 03:53:59.301837 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.301813    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh"
Sep 17 03:53:59.439976 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.439948    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkbgz\" (UniqueName: \"kubernetes.io/projected/b9c51d77-1a0c-41df-aa7a-43c46772e496-kube-api-access-pkbgz\") pod \"b9c51d77-1a0c-41df-aa7a-43c46772e496\" (UID: \"b9c51d77-1a0c-41df-aa7a-43c46772e496\") "
Sep 17 03:53:59.440180 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.439993    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c51d77-1a0c-41df-aa7a-43c46772e496-metrics-certs\") pod \"b9c51d77-1a0c-41df-aa7a-43c46772e496\" (UID: \"b9c51d77-1a0c-41df-aa7a-43c46772e496\") "
Sep 17 03:53:59.440180 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.440087    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c51d77-1a0c-41df-aa7a-43c46772e496-webhook-cert\") pod \"b9c51d77-1a0c-41df-aa7a-43c46772e496\" (UID: \"b9c51d77-1a0c-41df-aa7a-43c46772e496\") "
Sep 17 03:53:59.442019 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.441986    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c51d77-1a0c-41df-aa7a-43c46772e496-kube-api-access-pkbgz" (OuterVolumeSpecName: "kube-api-access-pkbgz") pod "b9c51d77-1a0c-41df-aa7a-43c46772e496" (UID: "b9c51d77-1a0c-41df-aa7a-43c46772e496"). InnerVolumeSpecName "kube-api-access-pkbgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:53:59.442129 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.442022    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c51d77-1a0c-41df-aa7a-43c46772e496-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b9c51d77-1a0c-41df-aa7a-43c46772e496" (UID: "b9c51d77-1a0c-41df-aa7a-43c46772e496"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:53:59.442129 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.442066    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c51d77-1a0c-41df-aa7a-43c46772e496-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b9c51d77-1a0c-41df-aa7a-43c46772e496" (UID: "b9c51d77-1a0c-41df-aa7a-43c46772e496"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:53:59.541356 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.541326    2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c51d77-1a0c-41df-aa7a-43c46772e496-webhook-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:59.541356 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.541355    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkbgz\" (UniqueName: \"kubernetes.io/projected/b9c51d77-1a0c-41df-aa7a-43c46772e496-kube-api-access-pkbgz\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:59.541356 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.541364    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c51d77-1a0c-41df-aa7a-43c46772e496-metrics-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:53:59.745461 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.745388    2557 generic.go:356] "Generic (PLEG): container finished" podID="b9c51d77-1a0c-41df-aa7a-43c46772e496" containerID="b64d21b3110e6e66fea9dacb8f14d429f9b4a54848935b967fe6748ffe5911ac" exitCode=0
Sep 17 03:53:59.745461 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.745427    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh"
Sep 17 03:53:59.745607 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.745475    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh" event={"ID":"b9c51d77-1a0c-41df-aa7a-43c46772e496","Type":"ContainerDied","Data":"b64d21b3110e6e66fea9dacb8f14d429f9b4a54848935b967fe6748ffe5911ac"}
Sep 17 03:53:59.745607 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.745526    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8c8d6b99-xc8jh" event={"ID":"b9c51d77-1a0c-41df-aa7a-43c46772e496","Type":"ContainerDied","Data":"bbecab56f70d6182554928c5c60eb3977f3c775f21e4261577021ecc01918a3e"}
Sep 17 03:53:59.745607 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.745547    2557 scope.go:117] "RemoveContainer" containerID="b64d21b3110e6e66fea9dacb8f14d429f9b4a54848935b967fe6748ffe5911ac"
Sep 17 03:53:59.754924 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.754903    2557 scope.go:117] "RemoveContainer" containerID="b64d21b3110e6e66fea9dacb8f14d429f9b4a54848935b967fe6748ffe5911ac"
Sep 17 03:53:59.755233 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:53:59.755186    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64d21b3110e6e66fea9dacb8f14d429f9b4a54848935b967fe6748ffe5911ac\": container with ID starting with b64d21b3110e6e66fea9dacb8f14d429f9b4a54848935b967fe6748ffe5911ac not found: ID does not exist" containerID="b64d21b3110e6e66fea9dacb8f14d429f9b4a54848935b967fe6748ffe5911ac"
Sep 17 03:53:59.755323 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.755241    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64d21b3110e6e66fea9dacb8f14d429f9b4a54848935b967fe6748ffe5911ac"} err="failed to get container status \"b64d21b3110e6e66fea9dacb8f14d429f9b4a54848935b967fe6748ffe5911ac\": rpc error: code = NotFound desc = could not find container \"b64d21b3110e6e66fea9dacb8f14d429f9b4a54848935b967fe6748ffe5911ac\": container with ID starting with b64d21b3110e6e66fea9dacb8f14d429f9b4a54848935b967fe6748ffe5911ac not found: ID does not exist"
Sep 17 03:53:59.766067 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.766040    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c8c8d6b99-xc8jh"]
Sep 17 03:53:59.769667 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.769639    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c8c8d6b99-xc8jh"]
Sep 17 03:53:59.914341 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:53:59.914316    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9c51d77-1a0c-41df-aa7a-43c46772e496" path="/var/lib/kubelet/pods/b9c51d77-1a0c-41df-aa7a-43c46772e496/volumes"
Sep 17 03:54:10.060388 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:54:10.060361    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:54:10.067011 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:54:10.066980    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:54:10.402068 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:54:10.402044    2557 scope.go:117] "RemoveContainer" containerID="73ab69dc258feaf4a250cb9051806df652cc6ac3a5cef7f57b3ac305be0d659d"
Sep 17 03:55:00.603760 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.603723    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c8c8d6b99-8lhd7"]
Sep 17 03:55:00.604235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604102    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a280dc9-dffc-4b55-a27e-4d8a093a8cd1" containerName="payload-pre-processing"
Sep 17 03:55:00.604235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604115    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a280dc9-dffc-4b55-a27e-4d8a093a8cd1" containerName="payload-pre-processing"
Sep 17 03:55:00.604235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604127    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52e7548a-c4f8-440a-9497-741ae9640730" containerName="maas-api"
Sep 17 03:55:00.604235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604136    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="52e7548a-c4f8-440a-9497-741ae9640730" containerName="maas-api"
Sep 17 03:55:00.604235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604150    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4e8d375-d954-4c6b-bb58-d88afd16cdd5" containerName="maas-api"
Sep 17 03:55:00.604235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604158    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4e8d375-d954-4c6b-bb58-d88afd16cdd5" containerName="maas-api"
Sep 17 03:55:00.604235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604174    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a831fc84-6101-417c-b3a0-ad97ea3df6e4" containerName="payload-processing"
Sep 17 03:55:00.604235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604182    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a831fc84-6101-417c-b3a0-ad97ea3df6e4" containerName="payload-processing"
Sep 17 03:55:00.604235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604209    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a" containerName="payload-processing"
Sep 17 03:55:00.604235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604218    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a" containerName="payload-processing"
Sep 17 03:55:00.604235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604231    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d634cb58-e29d-4bb5-9135-de8381a94411" containerName="istio-proxy"
Sep 17 03:55:00.604235 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604239    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d634cb58-e29d-4bb5-9135-de8381a94411" containerName="istio-proxy"
Sep 17 03:55:00.604707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604252    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9c51d77-1a0c-41df-aa7a-43c46772e496" containerName="manager"
Sep 17 03:55:00.604707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604261    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9c51d77-1a0c-41df-aa7a-43c46772e496" containerName="manager"
Sep 17 03:55:00.604707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604283    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d634cb58-e29d-4bb5-9135-de8381a94411" containerName="istio-proxy"
Sep 17 03:55:00.604707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604293    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d634cb58-e29d-4bb5-9135-de8381a94411" containerName="istio-proxy"
Sep 17 03:55:00.604707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604309    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e0dd334-39e0-4176-b826-fe45f297879f" containerName="payload-pre-processing"
Sep 17 03:55:00.604707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604317    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e0dd334-39e0-4176-b826-fe45f297879f" containerName="payload-pre-processing"
Sep 17 03:55:00.604707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604419    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4e8d375-d954-4c6b-bb58-d88afd16cdd5" containerName="maas-api"
Sep 17 03:55:00.604707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604431    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d634cb58-e29d-4bb5-9135-de8381a94411" containerName="istio-proxy"
Sep 17 03:55:00.604707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604437    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bfe9c51-5ca0-47c7-ad6a-5a3aa274318a" containerName="payload-processing"
Sep 17 03:55:00.604707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604443    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a831fc84-6101-417c-b3a0-ad97ea3df6e4" containerName="payload-processing"
Sep 17 03:55:00.604707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604449    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e0dd334-39e0-4176-b826-fe45f297879f" containerName="payload-pre-processing"
Sep 17 03:55:00.604707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604456    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d634cb58-e29d-4bb5-9135-de8381a94411" containerName="istio-proxy"
Sep 17 03:55:00.604707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604462    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a280dc9-dffc-4b55-a27e-4d8a093a8cd1" containerName="payload-pre-processing"
Sep 17 03:55:00.604707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604469    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="52e7548a-c4f8-440a-9497-741ae9640730" containerName="maas-api"
Sep 17 03:55:00.604707 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.604481    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9c51d77-1a0c-41df-aa7a-43c46772e496" containerName="manager"
Sep 17 03:55:00.607626 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.607610    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7"
Sep 17 03:55:00.610122 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.610091    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 17 03:55:00.610271 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.610116    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j9jwv\""
Sep 17 03:55:00.610271 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.610132    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 17 03:55:00.610271 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.610170    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 17 03:55:00.620564 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.616739    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c8c8d6b99-8lhd7"]
Sep 17 03:55:00.732395 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.732363    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-webhook-cert\") pod \"maas-controller-7c8c8d6b99-8lhd7\" (UID: \"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5\") " pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7"
Sep 17 03:55:00.732542 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.732401    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jspc7\" (UniqueName: \"kubernetes.io/projected/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-kube-api-access-jspc7\") pod \"maas-controller-7c8c8d6b99-8lhd7\" (UID: \"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5\") " pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7"
Sep 17 03:55:00.732542 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.732418    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-metrics-certs\") pod \"maas-controller-7c8c8d6b99-8lhd7\" (UID: \"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5\") " pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7"
Sep 17 03:55:00.833071 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.833046    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-webhook-cert\") pod \"maas-controller-7c8c8d6b99-8lhd7\" (UID: \"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5\") " pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7"
Sep 17 03:55:00.833214 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.833080    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jspc7\" (UniqueName: \"kubernetes.io/projected/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-kube-api-access-jspc7\") pod \"maas-controller-7c8c8d6b99-8lhd7\" (UID: \"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5\") " pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7"
Sep 17 03:55:00.833214 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.833096    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-metrics-certs\") pod \"maas-controller-7c8c8d6b99-8lhd7\" (UID: \"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5\") " pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7"
Sep 17 03:55:00.835510 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.835482    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-metrics-certs\") pod \"maas-controller-7c8c8d6b99-8lhd7\" (UID: \"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5\") " pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7"
Sep 17 03:55:00.835607 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.835509    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-webhook-cert\") pod \"maas-controller-7c8c8d6b99-8lhd7\" (UID: \"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5\") " pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7"
Sep 17 03:55:00.841547 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.841531    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jspc7\" (UniqueName: \"kubernetes.io/projected/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-kube-api-access-jspc7\") pod \"maas-controller-7c8c8d6b99-8lhd7\" (UID: \"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5\") " pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7"
Sep 17 03:55:00.925446 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:00.925422    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7"
Sep 17 03:55:01.195506 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:01.195474    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c8c8d6b99-8lhd7"]
Sep 17 03:55:01.196518 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:55:01.196487    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e4ef66_63ad_4bcf_bc92_be51a4d709c5.slice/crio-565404a0ce0709cf21698395fb284696ef78df466464b323827090e9abd93bd2 WatchSource:0}: Error finding container 565404a0ce0709cf21698395fb284696ef78df466464b323827090e9abd93bd2: Status 404 returned error can't find the container with id 565404a0ce0709cf21698395fb284696ef78df466464b323827090e9abd93bd2
Sep 17 03:55:01.197897 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:01.197877    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 03:55:02.073901 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:02.073804    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7" event={"ID":"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5","Type":"ContainerStarted","Data":"90b0d9d5e0e6539b856034d30fd1dabad63bef6d9f142a480db077dfd6567e67"}
Sep 17 03:55:02.073901 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:02.073870    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7" event={"ID":"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5","Type":"ContainerStarted","Data":"565404a0ce0709cf21698395fb284696ef78df466464b323827090e9abd93bd2"}
Sep 17 03:55:02.074339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:02.073985    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7"
Sep 17 03:55:02.090067 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:02.090007    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7" podStartSLOduration=1.605667332 podStartE2EDuration="2.089993259s" totalImagesPullingTime="484.325927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:55:02.088260527 +0000 UTC m=+2152.691356530" watchObservedRunningTime="2026-09-17 03:55:02.089993259 +0000 UTC m=+2152.693089279"
Sep 17 03:55:13.082024 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:13.081992    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7"
Sep 17 03:55:22.379007 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:22.378963    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c8c8d6b99-8lhd7"]
Sep 17 03:55:22.379547 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:22.379221    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7" podUID="a2e4ef66-63ad-4bcf-bc92-be51a4d709c5" containerName="manager" containerID="cri-o://90b0d9d5e0e6539b856034d30fd1dabad63bef6d9f142a480db077dfd6567e67" gracePeriod=30
Sep 17 03:55:22.628143 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:22.628117    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7"
Sep 17 03:55:22.803613 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:22.803539    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-metrics-certs\") pod \"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5\" (UID: \"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5\") "
Sep 17 03:55:22.803613 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:22.803579    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jspc7\" (UniqueName: \"kubernetes.io/projected/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-kube-api-access-jspc7\") pod \"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5\" (UID: \"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5\") "
Sep 17 03:55:22.803834 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:22.803638    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-webhook-cert\") pod \"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5\" (UID: \"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5\") "
Sep 17 03:55:22.805561 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:22.805529    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a2e4ef66-63ad-4bcf-bc92-be51a4d709c5" (UID: "a2e4ef66-63ad-4bcf-bc92-be51a4d709c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:55:22.805694 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:22.805635    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a2e4ef66-63ad-4bcf-bc92-be51a4d709c5" (UID: "a2e4ef66-63ad-4bcf-bc92-be51a4d709c5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:55:22.805739 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:22.805705    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-kube-api-access-jspc7" (OuterVolumeSpecName: "kube-api-access-jspc7") pod "a2e4ef66-63ad-4bcf-bc92-be51a4d709c5" (UID: "a2e4ef66-63ad-4bcf-bc92-be51a4d709c5"). InnerVolumeSpecName "kube-api-access-jspc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:55:22.904937 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:22.904910    2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-webhook-cert\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:55:22.904937 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:22.904933    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-metrics-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:55:22.905087 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:22.904942    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jspc7\" (UniqueName: \"kubernetes.io/projected/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5-kube-api-access-jspc7\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:55:23.180372 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.180336    2557 generic.go:356] "Generic (PLEG): container finished" podID="a2e4ef66-63ad-4bcf-bc92-be51a4d709c5" containerID="90b0d9d5e0e6539b856034d30fd1dabad63bef6d9f142a480db077dfd6567e67" exitCode=0
Sep 17 03:55:23.180500 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.180386    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7"
Sep 17 03:55:23.180500 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.180412    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7" event={"ID":"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5","Type":"ContainerDied","Data":"90b0d9d5e0e6539b856034d30fd1dabad63bef6d9f142a480db077dfd6567e67"}
Sep 17 03:55:23.180500 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.180470    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8c8d6b99-8lhd7" event={"ID":"a2e4ef66-63ad-4bcf-bc92-be51a4d709c5","Type":"ContainerDied","Data":"565404a0ce0709cf21698395fb284696ef78df466464b323827090e9abd93bd2"}
Sep 17 03:55:23.180675 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.180500    2557 scope.go:117] "RemoveContainer" containerID="90b0d9d5e0e6539b856034d30fd1dabad63bef6d9f142a480db077dfd6567e67"
Sep 17 03:55:23.189713 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.189696    2557 scope.go:117] "RemoveContainer" containerID="90b0d9d5e0e6539b856034d30fd1dabad63bef6d9f142a480db077dfd6567e67"
Sep 17 03:55:23.189980 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:55:23.189961    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b0d9d5e0e6539b856034d30fd1dabad63bef6d9f142a480db077dfd6567e67\": container with ID starting with 90b0d9d5e0e6539b856034d30fd1dabad63bef6d9f142a480db077dfd6567e67 not found: ID does not exist" containerID="90b0d9d5e0e6539b856034d30fd1dabad63bef6d9f142a480db077dfd6567e67"
Sep 17 03:55:23.190043 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.189986    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b0d9d5e0e6539b856034d30fd1dabad63bef6d9f142a480db077dfd6567e67"} err="failed to get container status \"90b0d9d5e0e6539b856034d30fd1dabad63bef6d9f142a480db077dfd6567e67\": rpc error: code = NotFound desc = could not find container \"90b0d9d5e0e6539b856034d30fd1dabad63bef6d9f142a480db077dfd6567e67\": container with ID starting with 90b0d9d5e0e6539b856034d30fd1dabad63bef6d9f142a480db077dfd6567e67 not found: ID does not exist"
Sep 17 03:55:23.200438 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.200416    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c8c8d6b99-8lhd7"]
Sep 17 03:55:23.206421 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.206400    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c8c8d6b99-8lhd7"]
Sep 17 03:55:23.915772 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.915745    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2e4ef66-63ad-4bcf-bc92-be51a4d709c5" path="/var/lib/kubelet/pods/a2e4ef66-63ad-4bcf-bc92-be51a4d709c5/volumes"
Sep 17 03:55:23.978640 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.978615    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c8c8d6b99-dtjhm"]
Sep 17 03:55:23.979034 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.979018    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2e4ef66-63ad-4bcf-bc92-be51a4d709c5" containerName="manager"
Sep 17 03:55:23.979082 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.979039    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2e4ef66-63ad-4bcf-bc92-be51a4d709c5" containerName="manager"
Sep 17 03:55:23.979168 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.979156    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2e4ef66-63ad-4bcf-bc92-be51a4d709c5" containerName="manager"
Sep 17 03:55:23.983871 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.983852    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm"
Sep 17 03:55:23.986036 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.986015    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 17 03:55:23.987050 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.987022    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 17 03:55:23.987321 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.987080    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 17 03:55:23.987321 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.987112    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j9jwv\""
Sep 17 03:55:23.992431 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:23.992403    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c8c8d6b99-dtjhm"]
Sep 17 03:55:24.114180 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:24.114147    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3acc54b4-1ff8-40a0-9191-8d419d2154fe-webhook-cert\") pod \"maas-controller-7c8c8d6b99-dtjhm\" (UID: \"3acc54b4-1ff8-40a0-9191-8d419d2154fe\") " pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm"
Sep 17 03:55:24.114317 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:24.114186    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3acc54b4-1ff8-40a0-9191-8d419d2154fe-metrics-certs\") pod \"maas-controller-7c8c8d6b99-dtjhm\" (UID: \"3acc54b4-1ff8-40a0-9191-8d419d2154fe\") " pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm"
Sep 17 03:55:24.114317 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:24.114305    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4mzl\" (UniqueName: \"kubernetes.io/projected/3acc54b4-1ff8-40a0-9191-8d419d2154fe-kube-api-access-c4mzl\") pod \"maas-controller-7c8c8d6b99-dtjhm\" (UID: \"3acc54b4-1ff8-40a0-9191-8d419d2154fe\") " pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm"
Sep 17 03:55:24.214939 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:24.214864    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4mzl\" (UniqueName: \"kubernetes.io/projected/3acc54b4-1ff8-40a0-9191-8d419d2154fe-kube-api-access-c4mzl\") pod \"maas-controller-7c8c8d6b99-dtjhm\" (UID: \"3acc54b4-1ff8-40a0-9191-8d419d2154fe\") " pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm"
Sep 17 03:55:24.215059 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:24.214977    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3acc54b4-1ff8-40a0-9191-8d419d2154fe-webhook-cert\") pod \"maas-controller-7c8c8d6b99-dtjhm\" (UID: \"3acc54b4-1ff8-40a0-9191-8d419d2154fe\") " pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm"
Sep 17 03:55:24.215059 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:24.215014    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3acc54b4-1ff8-40a0-9191-8d419d2154fe-metrics-certs\") pod \"maas-controller-7c8c8d6b99-dtjhm\" (UID: \"3acc54b4-1ff8-40a0-9191-8d419d2154fe\") " pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm"
Sep 17 03:55:24.217425 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:24.217402    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3acc54b4-1ff8-40a0-9191-8d419d2154fe-metrics-certs\") pod \"maas-controller-7c8c8d6b99-dtjhm\" (UID: \"3acc54b4-1ff8-40a0-9191-8d419d2154fe\") " pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm"
Sep 17 03:55:24.217522 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:24.217436    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3acc54b4-1ff8-40a0-9191-8d419d2154fe-webhook-cert\") pod \"maas-controller-7c8c8d6b99-dtjhm\" (UID: \"3acc54b4-1ff8-40a0-9191-8d419d2154fe\") " pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm"
Sep 17 03:55:24.222374 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:24.222355    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4mzl\" (UniqueName: \"kubernetes.io/projected/3acc54b4-1ff8-40a0-9191-8d419d2154fe-kube-api-access-c4mzl\") pod \"maas-controller-7c8c8d6b99-dtjhm\" (UID: \"3acc54b4-1ff8-40a0-9191-8d419d2154fe\") " pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm"
Sep 17 03:55:24.295879 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:24.295857    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm"
Sep 17 03:55:24.564417 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:24.564390    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c8c8d6b99-dtjhm"]
Sep 17 03:55:24.565906 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:55:24.565880    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acc54b4_1ff8_40a0_9191_8d419d2154fe.slice/crio-3bc8bf05c9494278765a2cfaf57c952f8ea3f7c02cf6e9b95c293e88cecd423c WatchSource:0}: Error finding container 3bc8bf05c9494278765a2cfaf57c952f8ea3f7c02cf6e9b95c293e88cecd423c: Status 404 returned error can't find the container with id 3bc8bf05c9494278765a2cfaf57c952f8ea3f7c02cf6e9b95c293e88cecd423c
Sep 17 03:55:25.200722 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:25.200638    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm" event={"ID":"3acc54b4-1ff8-40a0-9191-8d419d2154fe","Type":"ContainerStarted","Data":"7645eec0dbec5d7537726cf98f171ff9bf00556bf451db687361b24557acac8c"}
Sep 17 03:55:25.200722 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:25.200695    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm" event={"ID":"3acc54b4-1ff8-40a0-9191-8d419d2154fe","Type":"ContainerStarted","Data":"3bc8bf05c9494278765a2cfaf57c952f8ea3f7c02cf6e9b95c293e88cecd423c"}
Sep 17 03:55:25.201221 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:25.200779    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm"
Sep 17 03:55:25.216451 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:25.216403    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm" podStartSLOduration=1.878753839 podStartE2EDuration="2.216387101s" totalImagesPullingTime="337.633262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:55:25.215520215 +0000 UTC m=+2175.818616220" watchObservedRunningTime="2026-09-17 03:55:25.216387101 +0000 UTC m=+2175.819483106"
Sep 17 03:55:36.208388 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:55:36.208344    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c8c8d6b99-dtjhm"
Sep 17 03:59:10.091691 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:10.091658    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:59:10.099033 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:10.098995    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 03:59:39.992043 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:39.991957    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"]
Sep 17 03:59:39.995831 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:39.995805    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:40.004055 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:40.004026    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"]
Sep 17 03:59:40.099986 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:40.099959    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3d6fa6-e67d-4910-92f0-c32984075aa0-metrics-certs\") pod \"maas-api-7c7d4dbbc6-7nn4n\" (UID: \"dd3d6fa6-e67d-4910-92f0-c32984075aa0\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:40.100211 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:40.099997    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtcxz\" (UniqueName: \"kubernetes.io/projected/dd3d6fa6-e67d-4910-92f0-c32984075aa0-kube-api-access-xtcxz\") pod \"maas-api-7c7d4dbbc6-7nn4n\" (UID: \"dd3d6fa6-e67d-4910-92f0-c32984075aa0\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:40.100211 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:40.100022    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd3d6fa6-e67d-4910-92f0-c32984075aa0-maas-api-tls\") pod \"maas-api-7c7d4dbbc6-7nn4n\" (UID: \"dd3d6fa6-e67d-4910-92f0-c32984075aa0\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:40.100211 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:40.100139    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3d6fa6-e67d-4910-92f0-c32984075aa0-openshift-service-ca\") pod \"maas-api-7c7d4dbbc6-7nn4n\" (UID: \"dd3d6fa6-e67d-4910-92f0-c32984075aa0\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:40.201207 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:40.201173    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3d6fa6-e67d-4910-92f0-c32984075aa0-metrics-certs\") pod \"maas-api-7c7d4dbbc6-7nn4n\" (UID: \"dd3d6fa6-e67d-4910-92f0-c32984075aa0\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:40.201372 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:40.201267    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtcxz\" (UniqueName: \"kubernetes.io/projected/dd3d6fa6-e67d-4910-92f0-c32984075aa0-kube-api-access-xtcxz\") pod \"maas-api-7c7d4dbbc6-7nn4n\" (UID: \"dd3d6fa6-e67d-4910-92f0-c32984075aa0\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:40.201372 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:40.201328    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd3d6fa6-e67d-4910-92f0-c32984075aa0-maas-api-tls\") pod \"maas-api-7c7d4dbbc6-7nn4n\" (UID: \"dd3d6fa6-e67d-4910-92f0-c32984075aa0\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:40.201476 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:40.201402    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3d6fa6-e67d-4910-92f0-c32984075aa0-openshift-service-ca\") pod \"maas-api-7c7d4dbbc6-7nn4n\" (UID: \"dd3d6fa6-e67d-4910-92f0-c32984075aa0\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:40.202018 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:40.201994    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd3d6fa6-e67d-4910-92f0-c32984075aa0-openshift-service-ca\") pod \"maas-api-7c7d4dbbc6-7nn4n\" (UID: \"dd3d6fa6-e67d-4910-92f0-c32984075aa0\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:40.203612 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:40.203584    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd3d6fa6-e67d-4910-92f0-c32984075aa0-metrics-certs\") pod \"maas-api-7c7d4dbbc6-7nn4n\" (UID: \"dd3d6fa6-e67d-4910-92f0-c32984075aa0\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:40.203722 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:40.203688    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd3d6fa6-e67d-4910-92f0-c32984075aa0-maas-api-tls\") pod \"maas-api-7c7d4dbbc6-7nn4n\" (UID: \"dd3d6fa6-e67d-4910-92f0-c32984075aa0\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:40.208419 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:40.208395    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtcxz\" (UniqueName: \"kubernetes.io/projected/dd3d6fa6-e67d-4910-92f0-c32984075aa0-kube-api-access-xtcxz\") pod \"maas-api-7c7d4dbbc6-7nn4n\" (UID: \"dd3d6fa6-e67d-4910-92f0-c32984075aa0\") " pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:40.311705 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:40.311633    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:40.591759 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:40.591738    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"]
Sep 17 03:59:40.593542 ip-10-0-130-153 kubenswrapper[2557]: W0917 03:59:40.593517    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3d6fa6_e67d_4910_92f0_c32984075aa0.slice/crio-1477ac5ba435af159a690d1aaa7120252b551a60ba0e457421a4c721eff39c7a WatchSource:0}: Error finding container 1477ac5ba435af159a690d1aaa7120252b551a60ba0e457421a4c721eff39c7a: Status 404 returned error can't find the container with id 1477ac5ba435af159a690d1aaa7120252b551a60ba0e457421a4c721eff39c7a
Sep 17 03:59:41.491942 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:41.491904    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n" event={"ID":"dd3d6fa6-e67d-4910-92f0-c32984075aa0","Type":"ContainerStarted","Data":"a5fa20118c916ceb79c6a99416a81d3d5995a5f98239e23ba64784ecf5d58eb8"}
Sep 17 03:59:41.492328 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:41.491966    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n" event={"ID":"dd3d6fa6-e67d-4910-92f0-c32984075aa0","Type":"ContainerStarted","Data":"1477ac5ba435af159a690d1aaa7120252b551a60ba0e457421a4c721eff39c7a"}
Sep 17 03:59:41.492328 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:41.492005    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:41.508416 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:41.508345    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n" podStartSLOduration=2.140433956 podStartE2EDuration="2.508330994s" totalImagesPullingTime="367.897038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 03:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 03:59:41.506820362 +0000 UTC m=+2432.109916369" watchObservedRunningTime="2026-09-17 03:59:41.508330994 +0000 UTC m=+2432.111426997"
Sep 17 03:59:47.502685 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:47.502657    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c7d4dbbc6-7nn4n"
Sep 17 03:59:55.038376 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:55.037662    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"]
Sep 17 03:59:55.038376 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:55.037964    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh" podUID="47b28718-2cb2-473b-add7-d5e23b45b900" containerName="maas-api" containerID="cri-o://530aeaae4410c67750f5221a35a95c81c27600588a8eb654a1b212348edaa992" gracePeriod=30
Sep 17 03:59:56.287355 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.287332    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:59:56.330747 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.330717    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mkvv\" (UniqueName: \"kubernetes.io/projected/47b28718-2cb2-473b-add7-d5e23b45b900-kube-api-access-7mkvv\") pod \"47b28718-2cb2-473b-add7-d5e23b45b900\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") "
Sep 17 03:59:56.330852 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.330756    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b28718-2cb2-473b-add7-d5e23b45b900-maas-api-tls\") pod \"47b28718-2cb2-473b-add7-d5e23b45b900\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") "
Sep 17 03:59:56.330902 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.330853    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b28718-2cb2-473b-add7-d5e23b45b900-metrics-certs\") pod \"47b28718-2cb2-473b-add7-d5e23b45b900\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") "
Sep 17 03:59:56.330955 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.330931    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b28718-2cb2-473b-add7-d5e23b45b900-openshift-service-ca\") pod \"47b28718-2cb2-473b-add7-d5e23b45b900\" (UID: \"47b28718-2cb2-473b-add7-d5e23b45b900\") "
Sep 17 03:59:56.331426 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.331394    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b28718-2cb2-473b-add7-d5e23b45b900-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "47b28718-2cb2-473b-add7-d5e23b45b900" (UID: "47b28718-2cb2-473b-add7-d5e23b45b900"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 03:59:56.332948 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.332920    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b28718-2cb2-473b-add7-d5e23b45b900-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "47b28718-2cb2-473b-add7-d5e23b45b900" (UID: "47b28718-2cb2-473b-add7-d5e23b45b900"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:59:56.333228 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.333205    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b28718-2cb2-473b-add7-d5e23b45b900-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "47b28718-2cb2-473b-add7-d5e23b45b900" (UID: "47b28718-2cb2-473b-add7-d5e23b45b900"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 03:59:56.333348 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.333322    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b28718-2cb2-473b-add7-d5e23b45b900-kube-api-access-7mkvv" (OuterVolumeSpecName: "kube-api-access-7mkvv") pod "47b28718-2cb2-473b-add7-d5e23b45b900" (UID: "47b28718-2cb2-473b-add7-d5e23b45b900"). InnerVolumeSpecName "kube-api-access-7mkvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 03:59:56.432245 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.432223    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b28718-2cb2-473b-add7-d5e23b45b900-metrics-certs\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:59:56.432339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.432250    2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b28718-2cb2-473b-add7-d5e23b45b900-openshift-service-ca\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:59:56.432339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.432261    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mkvv\" (UniqueName: \"kubernetes.io/projected/47b28718-2cb2-473b-add7-d5e23b45b900-kube-api-access-7mkvv\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:59:56.432339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.432269    2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b28718-2cb2-473b-add7-d5e23b45b900-maas-api-tls\") on node \"ip-10-0-130-153.ec2.internal\" DevicePath \"\""
Sep 17 03:59:56.573245 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.573218    2557 generic.go:356] "Generic (PLEG): container finished" podID="47b28718-2cb2-473b-add7-d5e23b45b900" containerID="530aeaae4410c67750f5221a35a95c81c27600588a8eb654a1b212348edaa992" exitCode=0
Sep 17 03:59:56.573339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.573254    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"
Sep 17 03:59:56.573339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.573261    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh" event={"ID":"47b28718-2cb2-473b-add7-d5e23b45b900","Type":"ContainerDied","Data":"530aeaae4410c67750f5221a35a95c81c27600588a8eb654a1b212348edaa992"}
Sep 17 03:59:56.573339 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.573316    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh" event={"ID":"47b28718-2cb2-473b-add7-d5e23b45b900","Type":"ContainerDied","Data":"6c80537a65143d786cb860b7d244bbb2d4ef4413e4a0a2112b6ce9a663c925cb"}
Sep 17 03:59:56.573505 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.573347    2557 scope.go:117] "RemoveContainer" containerID="530aeaae4410c67750f5221a35a95c81c27600588a8eb654a1b212348edaa992"
Sep 17 03:59:56.582141 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.582124    2557 scope.go:117] "RemoveContainer" containerID="530aeaae4410c67750f5221a35a95c81c27600588a8eb654a1b212348edaa992"
Sep 17 03:59:56.582425 ip-10-0-130-153 kubenswrapper[2557]: E0917 03:59:56.582405    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"530aeaae4410c67750f5221a35a95c81c27600588a8eb654a1b212348edaa992\": container with ID starting with 530aeaae4410c67750f5221a35a95c81c27600588a8eb654a1b212348edaa992 not found: ID does not exist" containerID="530aeaae4410c67750f5221a35a95c81c27600588a8eb654a1b212348edaa992"
Sep 17 03:59:56.582486 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.582446    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"530aeaae4410c67750f5221a35a95c81c27600588a8eb654a1b212348edaa992"} err="failed to get container status \"530aeaae4410c67750f5221a35a95c81c27600588a8eb654a1b212348edaa992\": rpc error: code = NotFound desc = could not find container \"530aeaae4410c67750f5221a35a95c81c27600588a8eb654a1b212348edaa992\": container with ID starting with 530aeaae4410c67750f5221a35a95c81c27600588a8eb654a1b212348edaa992 not found: ID does not exist"
Sep 17 03:59:56.595586 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.595566    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"]
Sep 17 03:59:56.600301 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:56.600278    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5575f6cfb4-c6jvh"]
Sep 17 03:59:57.914738 ip-10-0-130-153 kubenswrapper[2557]: I0917 03:59:57.914704    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47b28718-2cb2-473b-add7-d5e23b45b900" path="/var/lib/kubelet/pods/47b28718-2cb2-473b-add7-d5e23b45b900/volumes"
Sep 17 04:00:16.566218 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:16.566160    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-grtpb_6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7/manager/0.log"
Sep 17 04:00:16.680982 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:16.680946    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tcntt_0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8/manager/0.log"
Sep 17 04:00:16.791167 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:16.791141    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c8c8d6b99-dtjhm_3acc54b4-1ff8-40a0-9191-8d419d2154fe/manager/0.log"
Sep 17 04:00:17.255090 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:17.255062    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6977f7b7bc-f8wcx_89bd3a1e-642e-4389-a7b4-e0407a99cd8e/copy-manifests/0.log"
Sep 17 04:00:17.261447 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:17.261420    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6977f7b7bc-f8wcx_89bd3a1e-642e-4389-a7b4-e0407a99cd8e/manager/0.log"
Sep 17 04:00:17.700116 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:17.700089    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c7d4dbbc6-7nn4n_dd3d6fa6-e67d-4910-92f0-c32984075aa0/maas-api/0.log"
Sep 17 04:00:18.222344 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:18.222314    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-jhbm5_078ee1f5-83c0-42a0-a1a8-012ca1341032/postgres/0.log"
Sep 17 04:00:19.267743 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:19.267706    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-8wh8n_8728ba23-f898-4af8-800d-13fc061c92c4/manager/0.log"
Sep 17 04:00:20.118769 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:20.118741    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww_1066f870-0e47-474b-a5bc-68795aa2f3a3/istio-proxy/0.log"
Sep 17 04:00:20.565376 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:20.565304    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-q9jxx_f1a4d22e-eade-4378-bdc4-5f42eb080071/istio-proxy/1.log"
Sep 17 04:00:20.695523 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:20.695495    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-j9c62_1dd3bf10-7094-4cb1-8a06-7dfbb62131ae/payload-pre-processing/0.log"
Sep 17 04:00:20.818660 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:20.818585    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7gscq_6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb/payload-processing/0.log"
Sep 17 04:00:20.923600 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:20.923561    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6685b49f78-sfnns_b1592c7c-2902-4b8d-9a15-b1bce854867f/router/0.log"
Sep 17 04:00:21.246947 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:21.246924    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-fsdx9_9378a674-d679-4414-ba82-0f364ba4a164/main/0.log"
Sep 17 04:00:21.577536 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:21.577461    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-glxxj_ce023e03-a8b3-4c4d-b46c-5dd04ef2434f/main/0.log"
Sep 17 04:00:21.694160 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:21.694132    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fg7qtc_ea85fde4-8a1c-4e42-9a32-1d799a3f0041/main/0.log"
Sep 17 04:00:29.813921 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:29.813894    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mc2fv_c1b8633f-f1cc-440b-96e6-7e0b526f6583/global-pull-secret-syncer/0.log"
Sep 17 04:00:29.916089 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:29.916061    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gg8lm_af354256-5b4f-4a41-8391-b8145ea28913/konnectivity-agent/0.log"
Sep 17 04:00:29.934879 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:29.934856    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-153.ec2.internal_be359c8ab8f51cc294084eab7fd5bdf6/haproxy/0.log"
Sep 17 04:00:34.807932 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:34.807898    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-8wh8n_8728ba23-f898-4af8-800d-13fc061c92c4/manager/0.log"
Sep 17 04:00:36.594389 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:36.594366    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rjm62_a130fd36-fe3a-47e3-a437-078ff2f08676/kube-state-metrics/0.log"
Sep 17 04:00:36.611835 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:36.611808    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rjm62_a130fd36-fe3a-47e3-a437-078ff2f08676/kube-rbac-proxy-main/0.log"
Sep 17 04:00:36.630499 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:36.630479    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rjm62_a130fd36-fe3a-47e3-a437-078ff2f08676/kube-rbac-proxy-self/0.log"
Sep 17 04:00:36.672030 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:36.672011    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-9b7d74d76-fnw2t_70e857a5-6ddf-4811-9b58-8ff90d0e27cc/metrics-server/0.log"
Sep 17 04:00:36.699936 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:36.699916    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-mw5fq_48f39267-6e56-4d92-9ce0-c14b039ebad9/monitoring-plugin/0.log"
Sep 17 04:00:36.730901 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:36.730884    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ptrr_3da63e2a-f87a-4fc6-b710-0848c4b89488/node-exporter/0.log"
Sep 17 04:00:36.751805 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:36.751788    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ptrr_3da63e2a-f87a-4fc6-b710-0848c4b89488/kube-rbac-proxy/0.log"
Sep 17 04:00:36.773036 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:36.773015    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6ptrr_3da63e2a-f87a-4fc6-b710-0848c4b89488/init-textfile/0.log"
Sep 17 04:00:36.990846 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:36.990813    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-vxqjh_a96ad582-ed86-424a-b88d-95f9ff7df8f6/kube-rbac-proxy-main/0.log"
Sep 17 04:00:37.019993 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:37.019976    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-vxqjh_a96ad582-ed86-424a-b88d-95f9ff7df8f6/kube-rbac-proxy-self/0.log"
Sep 17 04:00:37.073220 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:37.073188    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-vxqjh_a96ad582-ed86-424a-b88d-95f9ff7df8f6/openshift-state-metrics/0.log"
Sep 17 04:00:37.133751 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:37.133731    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cabbe6ab-13bc-4a45-938e-fdd788d9afbd/prometheus/0.log"
Sep 17 04:00:37.154784 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:37.154767    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cabbe6ab-13bc-4a45-938e-fdd788d9afbd/config-reloader/0.log"
Sep 17 04:00:37.173344 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:37.173327    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cabbe6ab-13bc-4a45-938e-fdd788d9afbd/thanos-sidecar/0.log"
Sep 17 04:00:37.189562 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:37.189545    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cabbe6ab-13bc-4a45-938e-fdd788d9afbd/kube-rbac-proxy-web/0.log"
Sep 17 04:00:37.205394 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:37.205378    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cabbe6ab-13bc-4a45-938e-fdd788d9afbd/kube-rbac-proxy/0.log"
Sep 17 04:00:37.239571 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:37.239518    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cabbe6ab-13bc-4a45-938e-fdd788d9afbd/kube-rbac-proxy-thanos/0.log"
Sep 17 04:00:37.259058 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:37.259042    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cabbe6ab-13bc-4a45-938e-fdd788d9afbd/init-config-reloader/0.log"
Sep 17 04:00:37.284176 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:37.284157    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-bqnhq_faba1bc0-5c90-49c7-9fbf-b2a81a4d3516/prometheus-operator/0.log"
Sep 17 04:00:37.303919 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:37.303901    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-bqnhq_faba1bc0-5c90-49c7-9fbf-b2a81a4d3516/kube-rbac-proxy/0.log"
Sep 17 04:00:37.325404 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:37.325386    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-7kkzg_a6bfc99c-f10f-452d-a620-c2cb9e34125e/prometheus-operator-admission-webhook/0.log"
Sep 17 04:00:39.474615 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:39.474552    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/2.log"
Sep 17 04:00:39.479837 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:39.479806    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvxrt_ba4e66fe-1155-4f15-841f-b5eadceec705/console-operator/3.log"
Sep 17 04:00:40.141089 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.141056    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"]
Sep 17 04:00:40.141508 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.141493    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b28718-2cb2-473b-add7-d5e23b45b900" containerName="maas-api"
Sep 17 04:00:40.141508 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.141509    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b28718-2cb2-473b-add7-d5e23b45b900" containerName="maas-api"
Sep 17 04:00:40.141615 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.141575    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b28718-2cb2-473b-add7-d5e23b45b900" containerName="maas-api"
Sep 17 04:00:40.145130 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.145110    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.147360 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.147335    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nv4hh\"/\"kube-root-ca.crt\""
Sep 17 04:00:40.147360 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.147350    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nv4hh\"/\"openshift-service-ca.crt\""
Sep 17 04:00:40.147832 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.147815    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nv4hh\"/\"default-dockercfg-8vt88\""
Sep 17 04:00:40.153704 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.153682    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"]
Sep 17 04:00:40.183452 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.183428    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/885a47f9-2a24-47de-a10d-c73c236b7ace-sys\") pod \"perf-node-gather-daemonset-rjswx\" (UID: \"885a47f9-2a24-47de-a10d-c73c236b7ace\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.183531 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.183453    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/885a47f9-2a24-47de-a10d-c73c236b7ace-proc\") pod \"perf-node-gather-daemonset-rjswx\" (UID: \"885a47f9-2a24-47de-a10d-c73c236b7ace\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.183531 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.183498    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/885a47f9-2a24-47de-a10d-c73c236b7ace-podres\") pod \"perf-node-gather-daemonset-rjswx\" (UID: \"885a47f9-2a24-47de-a10d-c73c236b7ace\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.183613 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.183560    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48zvk\" (UniqueName: \"kubernetes.io/projected/885a47f9-2a24-47de-a10d-c73c236b7ace-kube-api-access-48zvk\") pod \"perf-node-gather-daemonset-rjswx\" (UID: \"885a47f9-2a24-47de-a10d-c73c236b7ace\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.183653 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.183635    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/885a47f9-2a24-47de-a10d-c73c236b7ace-lib-modules\") pod \"perf-node-gather-daemonset-rjswx\" (UID: \"885a47f9-2a24-47de-a10d-c73c236b7ace\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.284396 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.284374    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/885a47f9-2a24-47de-a10d-c73c236b7ace-sys\") pod \"perf-node-gather-daemonset-rjswx\" (UID: \"885a47f9-2a24-47de-a10d-c73c236b7ace\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.284498 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.284401    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/885a47f9-2a24-47de-a10d-c73c236b7ace-proc\") pod \"perf-node-gather-daemonset-rjswx\" (UID: \"885a47f9-2a24-47de-a10d-c73c236b7ace\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.284498 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.284444    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/885a47f9-2a24-47de-a10d-c73c236b7ace-podres\") pod \"perf-node-gather-daemonset-rjswx\" (UID: \"885a47f9-2a24-47de-a10d-c73c236b7ace\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.284498 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.284468    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48zvk\" (UniqueName: \"kubernetes.io/projected/885a47f9-2a24-47de-a10d-c73c236b7ace-kube-api-access-48zvk\") pod \"perf-node-gather-daemonset-rjswx\" (UID: \"885a47f9-2a24-47de-a10d-c73c236b7ace\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.284661 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.284498    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/885a47f9-2a24-47de-a10d-c73c236b7ace-proc\") pod \"perf-node-gather-daemonset-rjswx\" (UID: \"885a47f9-2a24-47de-a10d-c73c236b7ace\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.284661 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.284504    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/885a47f9-2a24-47de-a10d-c73c236b7ace-lib-modules\") pod \"perf-node-gather-daemonset-rjswx\" (UID: \"885a47f9-2a24-47de-a10d-c73c236b7ace\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.284661 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.284504    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/885a47f9-2a24-47de-a10d-c73c236b7ace-sys\") pod \"perf-node-gather-daemonset-rjswx\" (UID: \"885a47f9-2a24-47de-a10d-c73c236b7ace\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.284661 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.284593    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/885a47f9-2a24-47de-a10d-c73c236b7ace-lib-modules\") pod \"perf-node-gather-daemonset-rjswx\" (UID: \"885a47f9-2a24-47de-a10d-c73c236b7ace\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.284661 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.284599    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/885a47f9-2a24-47de-a10d-c73c236b7ace-podres\") pod \"perf-node-gather-daemonset-rjswx\" (UID: \"885a47f9-2a24-47de-a10d-c73c236b7ace\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.292047 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.292029    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48zvk\" (UniqueName: \"kubernetes.io/projected/885a47f9-2a24-47de-a10d-c73c236b7ace-kube-api-access-48zvk\") pod \"perf-node-gather-daemonset-rjswx\" (UID: \"885a47f9-2a24-47de-a10d-c73c236b7ace\") " pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.456356 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.456300    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:40.724459 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.724435    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"]
Sep 17 04:00:40.726220 ip-10-0-130-153 kubenswrapper[2557]: W0917 04:00:40.726168    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod885a47f9_2a24_47de_a10d_c73c236b7ace.slice/crio-182f7ba83c24a5f1012f50e61b52f7465dfab66ec83ceaec5c500bde69c179a5 WatchSource:0}: Error finding container 182f7ba83c24a5f1012f50e61b52f7465dfab66ec83ceaec5c500bde69c179a5: Status 404 returned error can't find the container with id 182f7ba83c24a5f1012f50e61b52f7465dfab66ec83ceaec5c500bde69c179a5
Sep 17 04:00:40.797553 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:40.797527    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx" event={"ID":"885a47f9-2a24-47de-a10d-c73c236b7ace","Type":"ContainerStarted","Data":"182f7ba83c24a5f1012f50e61b52f7465dfab66ec83ceaec5c500bde69c179a5"}
Sep 17 04:00:41.805162 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:41.805114    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx" event={"ID":"885a47f9-2a24-47de-a10d-c73c236b7ace","Type":"ContainerStarted","Data":"50a26d728c1880e2b92788287b1c404ac9547577e9991d40d3954685b427ac4b"}
Sep 17 04:00:41.805671 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:41.805255    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:41.819430 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:41.819404    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4fqdd_7cbfac4e-1716-45d1-8a32-139bccea48a9/dns/0.log"
Sep 17 04:00:41.823065 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:41.823022    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx" podStartSLOduration=1.823001987 podStartE2EDuration="1.823001987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 04:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 04:00:41.821121015 +0000 UTC m=+2492.424217036" watchObservedRunningTime="2026-09-17 04:00:41.823001987 +0000 UTC m=+2492.426097991"
Sep 17 04:00:41.837097 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:41.837054    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4fqdd_7cbfac4e-1716-45d1-8a32-139bccea48a9/kube-rbac-proxy/0.log"
Sep 17 04:00:41.930784 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:41.930766    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b77f8_cf71bc53-8158-46f9-9ad2-9380beca6a48/dns-node-resolver/0.log"
Sep 17 04:00:42.502467 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:42.502439    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f6gqp_c9654da7-e49f-41b1-b764-41002e5bdb28/node-ca/0.log"
Sep 17 04:00:43.416130 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:43.416097    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww_1066f870-0e47-474b-a5bc-68795aa2f3a3/istio-proxy/0.log"
Sep 17 04:00:43.900630 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:43.900602    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-q9jxx_f1a4d22e-eade-4378-bdc4-5f42eb080071/istio-proxy/0.log"
Sep 17 04:00:43.903495 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:43.903477    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-q9jxx_f1a4d22e-eade-4378-bdc4-5f42eb080071/istio-proxy/1.log"
Sep 17 04:00:43.930507 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:43.930488    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-j9c62_1dd3bf10-7094-4cb1-8a06-7dfbb62131ae/payload-pre-processing/0.log"
Sep 17 04:00:43.962810 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:43.962785    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7gscq_6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb/payload-processing/0.log"
Sep 17 04:00:43.983369 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:43.983346    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6685b49f78-sfnns_b1592c7c-2902-4b8d-9a15-b1bce854867f/router/0.log"
Sep 17 04:00:44.857771 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:44.857731    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gm8dh_100b3459-13e3-4558-96ad-dd122ec25b08/serve-healthcheck-canary/0.log"
Sep 17 04:00:45.336748 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:45.336725    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2cxq9_18a22d55-ca40-44cb-87bc-2607b89f6385/kube-rbac-proxy/0.log"
Sep 17 04:00:45.353550 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:45.353524    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2cxq9_18a22d55-ca40-44cb-87bc-2607b89f6385/exporter/0.log"
Sep 17 04:00:45.369167 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:45.369145    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2cxq9_18a22d55-ca40-44cb-87bc-2607b89f6385/extractor/0.log"
Sep 17 04:00:47.572334 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:47.572309    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-grtpb_6ebefd1f-5f36-41ec-9fd3-ee0020d91fe7/manager/0.log"
Sep 17 04:00:47.819969 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:47.819938    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nv4hh/perf-node-gather-daemonset-rjswx"
Sep 17 04:00:47.839241 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:47.839147    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tcntt_0c369181-ccd7-4e5c-ab2e-bd3bbf87f0a8/manager/0.log"
Sep 17 04:00:47.883663 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:47.883640    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c8c8d6b99-dtjhm_3acc54b4-1ff8-40a0-9191-8d419d2154fe/manager/0.log"
Sep 17 04:00:48.116204 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:48.116179    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6977f7b7bc-f8wcx_89bd3a1e-642e-4389-a7b4-e0407a99cd8e/manager/0.log"
Sep 17 04:00:48.133941 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:48.133920    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6977f7b7bc-f8wcx_89bd3a1e-642e-4389-a7b4-e0407a99cd8e/copy-manifests/0.log"
Sep 17 04:00:49.361476 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:49.361444    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-55d557bdbb-lfwgd_b33ba81b-b815-434c-a131-6e57c834c613/manager/0.log"
Sep 17 04:00:50.107164 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:50.107126    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww_1066f870-0e47-474b-a5bc-68795aa2f3a3/istio-proxy/0.log"
Sep 17 04:00:50.563202 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:50.563164    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-q9jxx_f1a4d22e-eade-4378-bdc4-5f42eb080071/istio-proxy/0.log"
Sep 17 04:00:50.565962 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:50.565923    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-q9jxx_f1a4d22e-eade-4378-bdc4-5f42eb080071/istio-proxy/1.log"
Sep 17 04:00:50.594075 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:50.594050    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-j9c62_1dd3bf10-7094-4cb1-8a06-7dfbb62131ae/payload-pre-processing/0.log"
Sep 17 04:00:50.616456 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:50.616424    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7gscq_6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb/payload-processing/0.log"
Sep 17 04:00:50.632525 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:50.632503    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6685b49f78-sfnns_b1592c7c-2902-4b8d-9a15-b1bce854867f/router/0.log"
Sep 17 04:00:52.265338 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:52.265310    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-mq6ww_1066f870-0e47-474b-a5bc-68795aa2f3a3/istio-proxy/0.log"
Sep 17 04:00:52.728863 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:52.728782    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-q9jxx_f1a4d22e-eade-4378-bdc4-5f42eb080071/istio-proxy/0.log"
Sep 17 04:00:52.731594 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:52.731571    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-q9jxx_f1a4d22e-eade-4378-bdc4-5f42eb080071/istio-proxy/1.log"
Sep 17 04:00:52.758178 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:52.758138    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-j9c62_1dd3bf10-7094-4cb1-8a06-7dfbb62131ae/payload-pre-processing/0.log"
Sep 17 04:00:52.783916 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:52.783896    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7gscq_6aacdd63-3c26-4e1c-bd26-0a5a7cffcdcb/payload-processing/0.log"
Sep 17 04:00:52.807252 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:52.807229    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6685b49f78-sfnns_b1592c7c-2902-4b8d-9a15-b1bce854867f/router/0.log"
Sep 17 04:00:56.258101 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:56.258073    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l6v5r_8257d8ba-f173-4c72-b229-b9cc0326cf73/kube-multus-additional-cni-plugins/0.log"
Sep 17 04:00:56.274258 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:56.274229    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l6v5r_8257d8ba-f173-4c72-b229-b9cc0326cf73/egress-router-binary-copy/0.log"
Sep 17 04:00:56.290839 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:56.290805    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l6v5r_8257d8ba-f173-4c72-b229-b9cc0326cf73/cni-plugins/0.log"
Sep 17 04:00:56.309299 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:56.309275    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l6v5r_8257d8ba-f173-4c72-b229-b9cc0326cf73/bond-cni-plugin/0.log"
Sep 17 04:00:56.328106 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:56.328090    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l6v5r_8257d8ba-f173-4c72-b229-b9cc0326cf73/routeoverride-cni/0.log"
Sep 17 04:00:56.345704 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:56.345688    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l6v5r_8257d8ba-f173-4c72-b229-b9cc0326cf73/whereabouts-cni-bincopy/0.log"
Sep 17 04:00:56.362244 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:56.362221    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-l6v5r_8257d8ba-f173-4c72-b229-b9cc0326cf73/whereabouts-cni/0.log"
Sep 17 04:00:56.671299 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:56.671276    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hz26k_3a62682b-18c3-4dfe-982f-93d43396e8e7/kube-multus/0.log"
Sep 17 04:00:56.762465 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:56.762443    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5d8wb_c7146f5b-e9f5-4469-8161-756c03f69dac/network-metrics-daemon/0.log"
Sep 17 04:00:56.775363 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:56.775345    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5d8wb_c7146f5b-e9f5-4469-8161-756c03f69dac/kube-rbac-proxy/0.log"
Sep 17 04:00:58.002514 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:58.002488    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkpkk_386c6867-0049-47d3-90ac-e66fdb3ae814/ovn-controller/0.log"
Sep 17 04:00:58.019011 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:58.018990    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkpkk_386c6867-0049-47d3-90ac-e66fdb3ae814/ovn-acl-logging/0.log"
Sep 17 04:00:58.037451 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:58.037429    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkpkk_386c6867-0049-47d3-90ac-e66fdb3ae814/kube-rbac-proxy-node/0.log"
Sep 17 04:00:58.054510 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:58.054491    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkpkk_386c6867-0049-47d3-90ac-e66fdb3ae814/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 04:00:58.072881 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:58.072865    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkpkk_386c6867-0049-47d3-90ac-e66fdb3ae814/northd/0.log"
Sep 17 04:00:58.089924 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:58.089907    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkpkk_386c6867-0049-47d3-90ac-e66fdb3ae814/nbdb/0.log"
Sep 17 04:00:58.108232 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:58.108213    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkpkk_386c6867-0049-47d3-90ac-e66fdb3ae814/sbdb/0.log"
Sep 17 04:00:58.167442 ip-10-0-130-153 kubenswrapper[2557]: I0917 04:00:58.167418    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkpkk_386c6867-0049-47d3-90ac-e66fdb3ae814/ovnkube-controller/0.log"