Sep 12 00:42:54.966895 ip-10-0-136-121 systemd[1]: Starting Kubernetes Kubelet...
Sep 12 00:42:55.400530 ip-10-0-136-121 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 00:42:55.400530 ip-10-0-136-121 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 12 00:42:55.400530 ip-10-0-136-121 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 00:42:55.400530 ip-10-0-136-121 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 12 00:42:55.401131 ip-10-0-136-121 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 00:42:55.401528 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.401429    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 12 00:42:55.406750 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406733    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:42:55.406750 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406749    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406755    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406759    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406762    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406766    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406769    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406772    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406775    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406778    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406781    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406783    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406786    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406788    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406795    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406798    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406800    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406803    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406806    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406809    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406811    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:42:55.406816 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406814    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406816    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406819    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406823    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406825    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406828    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406831    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406833    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406836    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406839    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406841    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406844    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406847    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406849    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406852    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406854    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406857    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406859    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406862    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:42:55.407279 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406864    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406866    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406869    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406871    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406874    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406877    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406880    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406883    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406886    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406888    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406890    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406893    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406895    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406898    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406901    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406904    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406907    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406909    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406912    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:42:55.407735 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406914    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406917    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406919    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406922    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406925    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406927    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406931    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406934    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406937    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406940    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406944    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406947    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406950    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406952    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406955    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406958    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406961    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406964    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406966    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406969    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:42:55.408217 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406972    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406975    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406978    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406981    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406984    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406986    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406989    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406991    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406995    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.406998    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.407001    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.407003    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.407006    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.407008    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.407011    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.407013    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.407016    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.407018    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.407021    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.407023    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:42:55.408696 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408446    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408454    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408457    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408461    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408463    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408466    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408469    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408471    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408474    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408477    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408479    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408482    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408487    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408490    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408493    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408495    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408498    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408500    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408503    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:42:55.409275 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408505    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408509    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408512    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408514    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408517    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408520    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408522    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408525    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408527    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408530    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408535    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408538    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408541    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408544    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408547    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408549    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408552    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408557    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408560    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:42:55.410022 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408562    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408565    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408568    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408570    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408573    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408575    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408578    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408581    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408584    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408586    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408589    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408591    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408594    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408596    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408613    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408616    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408619    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408621    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408624    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408627    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:42:55.410547 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408629    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408633    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408635    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408638    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408641    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408643    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408646    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408648    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408651    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408653    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408656    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408658    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408661    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408663    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408666    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408668    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408671    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408673    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408676    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408679    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:42:55.411046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408682    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408684    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408687    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408690    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408692    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408695    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408698    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408701    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408704    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408706    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408709    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408712    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408714    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408717    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408719    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408722    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408725    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408727    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408730    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408732    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:42:55.411530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.408735    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409591    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409620    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409632    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409637    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409643    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409648    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409658    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409662    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409666    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409670    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409675    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409683    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409689    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409694    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409699    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409704    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409709    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409714    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:42:55.412027 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409718    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409728    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409733    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409737    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409741    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409746    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409750    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409754    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409759    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409763    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409767    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409772    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409776    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409785    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409791    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409797    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409802    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409807    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409811    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:42:55.412482 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409819    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409824    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409829    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409833    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409838    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409842    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409851    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409856    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409861    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409865    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409869    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409873    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409878    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409882    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409886    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409891    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409895    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409899    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409907    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409913    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:42:55.412952 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409917    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409932    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409935    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409938    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409941    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409944    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409947    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409950    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409952    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409955    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409958    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409961    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409964    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409966    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409973    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409978    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409983    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409986    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409990    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.409993    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:42:55.413434 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410000    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410024    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410044    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410136    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410147    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410152    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410155    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410160    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410164    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410168    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410171    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410175    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410179    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410183    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410187    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410191    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410195    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410198    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410202    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410206    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:42:55.413926 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410210    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410215    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410303    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410339    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410346    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410350    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410355    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410358    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410363    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410367    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410370    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410374    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410377    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410381    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410384    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410387    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410392    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410395    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410397    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410400    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410404    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410407    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410410    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410413    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 12 00:42:55.414412 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410417    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410420    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410423    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410426    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410429    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410432    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410435    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410438    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410441    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410444    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410448    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410453    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410456    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410459    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410462    2579 flags.go:64] FLAG: --enable-server="true"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410465    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410470    2579 flags.go:64] FLAG: --event-burst="100"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410473    2579 flags.go:64] FLAG: --event-qps="50"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410476    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410479    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410481    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410485    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410488    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410491    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410495    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 12 00:42:55.414994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410498    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410501    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410504    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410506    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410509    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410512    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410515    2579 flags.go:64] FLAG: --feature-gates=""
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410518    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410521    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410524    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410527    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410530    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410533    2579 flags.go:64] FLAG: --help="false"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410536    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410539    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410542    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410546    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410549    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410555    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410558    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410561    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410564    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410566    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 12 00:42:55.415578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410569    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410572    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410576    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410578    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410581    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410584    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410587    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410590    2579 flags.go:64] FLAG: --lock-file=""
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410594    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410614    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410620    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410627    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410631    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410633    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410636    2579 flags.go:64] FLAG: --logging-format="text"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410639    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410642    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410645    2579 flags.go:64] FLAG: --manifest-url=""
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410648    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410652    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410655    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410659    2579 flags.go:64] FLAG: --max-pods="110"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410662    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410665    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410668    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 12 00:42:55.416193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410671    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410674    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410679    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410682    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410689    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410692    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410695    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410698    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410701    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410706    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410709    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410712    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410715    2579 flags.go:64] FLAG: --port="10250"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410718    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410721    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0419ee655adaa1763"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410724    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410729    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410732    2579 flags.go:64] FLAG: --register-node="true"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410735    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410739    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410741    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410745    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410748    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410752    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410755    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 12 00:42:55.416802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410758    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410760    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410763    2579 flags.go:64] FLAG: --runonce="false"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410766    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410769    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410772    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410775    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410777    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410780    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410785    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410788    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410791    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410794    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410796    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410799    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410802    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410805    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410808    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410813    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410816    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410819    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410823    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410826    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410830    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410833    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 12 00:42:55.417418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410836    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410839    2579 flags.go:64] FLAG: --v="2"
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410845    2579 flags.go:64] FLAG: --version="false"
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410849    2579 flags.go:64] FLAG: --vmodule=""
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410853    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.410856    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410980    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410984    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410987    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410990    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410993    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410996    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.410999    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411001    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411004    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411007    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411011    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411013    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411016    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411018    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:42:55.418063 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411021    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411025    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411029    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411032    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411035    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411038    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411041    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411044    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411046    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411049    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411054    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411057    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411059    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411062    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411065    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411067    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411070    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411073    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411076    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:42:55.418570 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411079    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411082    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411084    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411087    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411090    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411092    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411095    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411098    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411100    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411105    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411107    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411110    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411112    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411115    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411118    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411120    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411123    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411125    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411128    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411132    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:42:55.419046 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411135    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411138    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411142    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411147    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411150    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411153    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411156    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411159    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411161    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411164    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411167    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411170    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411172    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411174    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411177    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411180    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411182    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411185    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411187    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:42:55.419562 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411190    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411193    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411197    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411199    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411202    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411204    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411207    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411209    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411212    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411214    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411217    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411219    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411222    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411224    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411227    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411238    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411242    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411244    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411247    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411250    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:42:55.420059 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411252    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:42:55.420537 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411255    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:42:55.420537 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411257    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:42:55.420537 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411260    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:42:55.420537 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411262    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:42:55.420537 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411265    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:42:55.420537 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.411268    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:42:55.420537 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.411274    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 00:42:55.420537 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.419180    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 12 00:42:55.420537 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.419196    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 12 00:42:55.420537 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.419215    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 12 00:42:55.420537 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.419220    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 12 00:42:55.420537 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419277    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:42:55.420537 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419282    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:42:55.420537 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419285    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:42:55.420537 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419289    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419292    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419295    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419298    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419301    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419304    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419307    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419310    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419312    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419315    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419318    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419320    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419323    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419326    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419329    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419331    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419334    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419336    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419339    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419342    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:42:55.420934 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419344    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419347    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419349    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419352    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419355    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419357    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419363    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419367    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419369    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419372    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419375    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419377    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419380    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419382    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419385    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419388    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419390    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419392    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419395    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:42:55.421412 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419398    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419400    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419403    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419405    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419408    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419411    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419413    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419416    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419418    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419420    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419423    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419426    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419428    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419431    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419433    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419436    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419438    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419441    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419443    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419447    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:42:55.421889 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419450    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419453    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419455    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419458    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419461    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419463    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419466    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419468    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419471    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419473    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419475    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419478    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419480    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419483    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419485    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419487    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419490    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419492    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419495    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419497    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:42:55.422371 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419500    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419502    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419505    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419507    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419510    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419512    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419515    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419518    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419522    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419526    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419529    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419532    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419535    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419538    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419541    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419544    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:42:55.422940 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419546    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:42:55.423325 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.419551    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 00:42:55.423325 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419687    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 00:42:55.423325 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419692    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 00:42:55.423325 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419696    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 00:42:55.423325 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419698    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 00:42:55.423325 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419701    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 00:42:55.423325 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419703    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 00:42:55.423325 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419706    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 00:42:55.423325 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419709    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 00:42:55.423325 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419711    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 00:42:55.423325 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419714    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 00:42:55.423325 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419716    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 00:42:55.423325 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419719    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 00:42:55.423325 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419721    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 00:42:55.423325 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419724    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419726    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419729    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419731    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419734    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419737    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419740    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419743    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419745    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419748    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419751    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419753    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419757    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419761    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419764    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419767    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419770    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419773    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419776    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 00:42:55.423799 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419779    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419782    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419784    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419787    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419790    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419792    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419795    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419797    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419800    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419803    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419805    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419808    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419811    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419813    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419816    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419818    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419821    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419823    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419826    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 00:42:55.424255 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419828    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419831    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419833    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419836    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419838    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419841    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419843    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419846    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419848    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419851    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419854    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419857    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419859    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419862    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419864    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419867    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419869    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419872    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419874    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419877    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 00:42:55.424730 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419881    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419884    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419886    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419889    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419894    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419897    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419900    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419902    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419905    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419907    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419910    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419912    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419915    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419917    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419920    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419922    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419924    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419927    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419930    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419933    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 00:42:55.425304 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419936    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 00:42:55.425794 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419938    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 00:42:55.425794 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419941    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 00:42:55.425794 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419945    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 00:42:55.425794 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419948    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 00:42:55.425794 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419950    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 00:42:55.425794 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419953    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 00:42:55.425794 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:55.419955    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 00:42:55.425794 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.419960    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 00:42:55.425794 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.420078    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 12 00:42:55.425794 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.422351    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 12 00:42:55.425794 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.423382    2579 server.go:1019] "Starting client certificate rotation"
Sep 12 00:42:55.425794 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.423474    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 12 00:42:55.425794 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.423539    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 12 00:42:55.426487 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.426474    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 12 00:42:55.427759 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.427739    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 12 00:42:55.437903 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.437883    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 12 00:42:55.441082 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.441066    2579 log.go:25] "Validated CRI v1 image API"
Sep 12 00:42:55.441986 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.441970    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 12 00:42:55.446041 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.446025    2579 fs.go:135] Filesystem UUIDs: map[38c8d8a6-ed4a-433e-b7b3-86986e5bb519:/dev/nvme0n1p4 4a9c999e-e96d-49dd-a192-3ef93e83bca1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 12 00:42:55.446108 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.446041    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 12 00:42:55.451125 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.450923    2579 manager.go:216] Machine: {Timestamp:2026-09-12 00:42:55.449501673 +0000 UTC m=+0.368793459 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094260 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2da87ecc17afce36e610a0d70de4bf SystemUUID:ec2da87e-cc17-afce-36e6-10a0d70de4bf BootID:b565ddcc-0044-462b-825d-e421c2620d74 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:65:46:a5:a3:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:65:46:a5:a3:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:03:e2:54:23:2a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 12 00:42:55.451125 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.451121    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 12 00:42:55.451219 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.451193    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 12 00:42:55.454570 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.454547    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 12 00:42:55.454721 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.454573    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-121.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 12 00:42:55.455474 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.455465    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 12 00:42:55.455504 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.455477    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 12 00:42:55.455504 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.455484    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 12 00:42:55.455504 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.455497    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 12 00:42:55.455802 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.455788    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 12 00:42:55.456654 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.456644    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 12 00:42:55.461001 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.460989    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 12 00:42:55.461113 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.461103    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 12 00:42:55.461142 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.461131    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 12 00:42:55.461142 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.461139    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 12 00:42:55.461198 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.461155    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 12 00:42:55.461198 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.461164    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 12 00:42:55.461198 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.461178    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 12 00:42:55.461517 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.461507    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 12 00:42:55.461561 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.461524    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 12 00:42:55.465371 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.465355    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 12 00:42:55.469193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.469174    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 12 00:42:55.469264 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.469208    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 12 00:42:55.471044 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.471023    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 12 00:42:55.471044 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.471047    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 12 00:42:55.471151 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.471059    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 12 00:42:55.471151 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.471070    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 12 00:42:55.471151 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.471088    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 12 00:42:55.471151 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.471096    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 12 00:42:55.471151 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.471114    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 12 00:42:55.471151 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.471122    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 12 00:42:55.471151 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.471151    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 12 00:42:55.471151 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.471158    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 12 00:42:55.471398 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.471170    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 12 00:42:55.471398 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.471175    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 12 00:42:55.471398 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.471202    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 12 00:42:55.471398 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.471212    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 12 00:42:55.471498 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.471387    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 12 00:42:55.471528 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.471492    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 12 00:42:55.474808 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.474796    2579 server.go:1268] "Started kubelet"
Sep 12 00:42:55.474884 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.474858    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 12 00:42:55.475330 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.474957    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 12 00:42:55.475387 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.475356    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 12 00:42:55.475656 ip-10-0-136-121 systemd[1]: Started Kubernetes Kubelet.
Sep 12 00:42:55.480007 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.476673    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 12 00:42:55.480007 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.479992    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 12 00:42:55.483849 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.483828    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 12 00:42:55.485707 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.485691    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-121.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 12 00:42:55.486149 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.486135    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 12 00:42:55.486746 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.485628    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-121.ec2.internal.18d46c11c09e48a3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-121.ec2.internal,UID:ip-10-0-136-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-121.ec2.internal,},FirstTimestamp:2026-09-12 00:42:55.474772131 +0000 UTC m=+0.394063909,LastTimestamp:2026-09-12 00:42:55.474772131 +0000 UTC m=+0.394063909,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-121.ec2.internal,}"
Sep 12 00:42:55.486824 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.486810    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 12 00:42:55.486942 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.486924    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:55.487615 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.487585    2579 factory.go:55] Registering systemd factory
Sep 12 00:42:55.487710 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.487641    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 12 00:42:55.487710 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.487609    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 12 00:42:55.487710 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.487595    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 12 00:42:55.487710 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.487689    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 12 00:42:55.487942 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.487783    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 12 00:42:55.487942 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.487792    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 12 00:42:55.487942 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.487842    2579 factory.go:153] Registering CRI-O factory
Sep 12 00:42:55.487942 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.487865    2579 factory.go:223] Registration of the crio container factory successfully
Sep 12 00:42:55.487942 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.487910    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 12 00:42:55.487942 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.487923    2579 factory.go:103] Registering Raw factory
Sep 12 00:42:55.487942 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.487935    2579 manager.go:1196] Started watching for new ooms in manager
Sep 12 00:42:55.488310 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.488299    2579 manager.go:349] Starting recovery of all containers
Sep 12 00:42:55.492328 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.492304    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 12 00:42:55.492439 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.492401    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 12 00:42:55.497783 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.497768    2579 manager.go:354] Recovery completed
Sep 12 00:42:55.502848 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.502834    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:42:55.504443 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.504419    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:42:55.504517 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.504460    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:42:55.504517 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.504473    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:42:55.504990 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.504976    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 12 00:42:55.504990 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.504990    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 12 00:42:55.505145 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.505007    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 12 00:42:55.505145 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.504983    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:42:55.505837 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.505821    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:42:55.505913 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.505850    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:42:55.505913 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.505860    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:42:55.506811 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.506531    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-121.ec2.internal.18d46c11c263093d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-121.ec2.internal,UID:ip-10-0-136-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-121.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-121.ec2.internal,},FirstTimestamp:2026-09-12 00:42:55.504443709 +0000 UTC m=+0.423735488,LastTimestamp:2026-09-12 00:42:55.504443709 +0000 UTC m=+0.423735488,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-121.ec2.internal,}"
Sep 12 00:42:55.513860 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.513842    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-48vzh"
Sep 12 00:42:55.518086 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.518011    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-121.ec2.internal.18d46c11c26363c7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-121.ec2.internal,UID:ip-10-0-136-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-121.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-121.ec2.internal,},FirstTimestamp:2026-09-12 00:42:55.504466887 +0000 UTC m=+0.423758666,LastTimestamp:2026-09-12 00:42:55.504466887 +0000 UTC m=+0.423758666,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-121.ec2.internal,}"
Sep 12 00:42:55.521983 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.521968    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-48vzh"
Sep 12 00:42:55.529841 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.529784    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-121.ec2.internal.18d46c11c26391fd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-121.ec2.internal,UID:ip-10-0-136-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-121.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-121.ec2.internal,},FirstTimestamp:2026-09-12 00:42:55.504478717 +0000 UTC m=+0.423770497,LastTimestamp:2026-09-12 00:42:55.504478717 +0000 UTC m=+0.423770497,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-121.ec2.internal,}"
Sep 12 00:42:55.540551 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.540474    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-121.ec2.internal.18d46c11c263093d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-121.ec2.internal.18d46c11c263093d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-121.ec2.internal,UID:ip-10-0-136-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-121.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-121.ec2.internal,},FirstTimestamp:2026-09-12 00:42:55.504443709 +0000 UTC m=+0.423735488,LastTimestamp:2026-09-12 00:42:55.505836273 +0000 UTC m=+0.425128055,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-121.ec2.internal,}"
Sep 12 00:42:55.565304 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.550351    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-121.ec2.internal.18d46c11c26363c7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-121.ec2.internal.18d46c11c26363c7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-121.ec2.internal,UID:ip-10-0-136-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-121.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-121.ec2.internal,},FirstTimestamp:2026-09-12 00:42:55.504466887 +0000 UTC m=+0.423758666,LastTimestamp:2026-09-12 00:42:55.505855032 +0000 UTC m=+0.425146811,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-121.ec2.internal,}"
Sep 12 00:42:55.565304 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.561367    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-121.ec2.internal.18d46c11c26391fd\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-121.ec2.internal.18d46c11c26391fd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-121.ec2.internal,UID:ip-10-0-136-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-121.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-121.ec2.internal,},FirstTimestamp:2026-09-12 00:42:55.504478717 +0000 UTC m=+0.423770497,LastTimestamp:2026-09-12 00:42:55.505865247 +0000 UTC m=+0.425157024,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-121.ec2.internal,}"
Sep 12 00:42:55.583373 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.583349    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 12 00:42:55.584447 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.584431    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 12 00:42:55.584540 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.584479    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 12 00:42:55.584540 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.584499    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 12 00:42:55.584641 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.584538    2579 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 12 00:42:55.587010 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.586984    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:55.594866 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.594846    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 12 00:42:55.649360 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.649341    2579 policy_none.go:49] "None policy: Start"
Sep 12 00:42:55.649448 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.649375    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 12 00:42:55.649448 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.649386    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 12 00:42:55.651557 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.651504    2579 policy_none.go:47] "Start"
Sep 12 00:42:55.682651 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.682637    2579 manager.go:337] "Starting Device Plugin manager"
Sep 12 00:42:55.682715 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.682670    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 12 00:42:55.682715 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.682683    2579 server.go:87] "Starting device plugin registration server"
Sep 12 00:42:55.683027 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.683016    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 12 00:42:55.683098 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.683031    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 12 00:42:55.683168 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.683149    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 12 00:42:55.683246 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.683234    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 12 00:42:55.683299 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.683247    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 12 00:42:55.683735 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.683719    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 12 00:42:55.683820 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.683770    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:55.684892 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.684879    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal"]
Sep 12 00:42:55.684935 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.684930    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:42:55.686229 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.686216    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:42:55.686295 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.686241    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:42:55.686295 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.686257    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:42:55.688491 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.688479    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:42:55.688592 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.688579    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.688675 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.688621    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:42:55.688675 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.688632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd4b87aecbc86ab2cf93f2fecaacbc0b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal\" (UID: \"dd4b87aecbc86ab2cf93f2fecaacbc0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.688675 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.688657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd4b87aecbc86ab2cf93f2fecaacbc0b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal\" (UID: \"dd4b87aecbc86ab2cf93f2fecaacbc0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.689191 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.689174    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:42:55.689254 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.689205    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:42:55.689254 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.689216    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:42:55.689317 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.689179    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:42:55.689317 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.689278    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:42:55.689317 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.689291    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:42:55.691451 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.691438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.691492 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.691460    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:42:55.692064 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.692051    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:42:55.692119 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.692071    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:42:55.692119 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.692082    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:42:55.697785 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.697771    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-121.ec2.internal\" not found" node="ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.735138 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.735116    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-121.ec2.internal\" not found" node="ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.739171 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.739155    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-121.ec2.internal\" not found" node="ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.783553 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.783538    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:42:55.784332 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.784316    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:42:55.784398 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.784349    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:42:55.784398 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.784362    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:42:55.784398 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.784386    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.789418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.789401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7cbe291fdf0e33f3b3d11babbfc8736a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-121.ec2.internal\" (UID: \"7cbe291fdf0e33f3b3d11babbfc8736a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.789495 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.789440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd4b87aecbc86ab2cf93f2fecaacbc0b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal\" (UID: \"dd4b87aecbc86ab2cf93f2fecaacbc0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.789495 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.789470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd4b87aecbc86ab2cf93f2fecaacbc0b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal\" (UID: \"dd4b87aecbc86ab2cf93f2fecaacbc0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.789581 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.789521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd4b87aecbc86ab2cf93f2fecaacbc0b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal\" (UID: \"dd4b87aecbc86ab2cf93f2fecaacbc0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.789581 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.789524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd4b87aecbc86ab2cf93f2fecaacbc0b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal\" (UID: \"dd4b87aecbc86ab2cf93f2fecaacbc0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.794282 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.794270    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.794332 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.794287    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-121.ec2.internal\": node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:55.812864 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.812843    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:55.890145 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.890121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7cbe291fdf0e33f3b3d11babbfc8736a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-121.ec2.internal\" (UID: \"7cbe291fdf0e33f3b3d11babbfc8736a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.890238 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:55.890189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7cbe291fdf0e33f3b3d11babbfc8736a-config\") pod \"kube-apiserver-proxy-ip-10-0-136-121.ec2.internal\" (UID: \"7cbe291fdf0e33f3b3d11babbfc8736a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal"
Sep 12 00:42:55.913274 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:55.913231    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:56.014025 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:56.013992    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:56.037162 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:56.037136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 12 00:42:56.042152 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:56.041677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal"
Sep 12 00:42:56.114413 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:56.114388    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:56.214891 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:56.214847    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:56.315230 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:56.315207    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:56.317726 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:56.317713    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:42:56.416285 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:56.416263    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:56.423432 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:56.423413    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 12 00:42:56.423545 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:56.423530    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 12 00:42:56.517036 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:56.516983    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:56.582110 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:56.582088    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 00:42:56.583364 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:56.583342    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 00:42:56.583465 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:56.583379    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 00:42:56.583465 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:56.583395    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeHasSufficientPID"
Sep 12 00:42:56.607964 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:56.607814    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4b87aecbc86ab2cf93f2fecaacbc0b.slice/crio-69120cc37d3b04929912cb6e48c43106b93aa3f33f5a5c490a7366ca0889568f WatchSource:0}: Error finding container 69120cc37d3b04929912cb6e48c43106b93aa3f33f5a5c490a7366ca0889568f: Status 404 returned error can't find the container with id 69120cc37d3b04929912cb6e48c43106b93aa3f33f5a5c490a7366ca0889568f
Sep 12 00:42:56.608716 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:56.608698    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbe291fdf0e33f3b3d11babbfc8736a.slice/crio-7997e69c0e8c0d44e6ee97608af037036448c6aa2570d4e7bed368d394fdf46f WatchSource:0}: Error finding container 7997e69c0e8c0d44e6ee97608af037036448c6aa2570d4e7bed368d394fdf46f: Status 404 returned error can't find the container with id 7997e69c0e8c0d44e6ee97608af037036448c6aa2570d4e7bed368d394fdf46f
Sep 12 00:42:56.614319 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:56.614300    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:42:56.618055 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:56.618039    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:56.653211 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:56.653183    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-11 00:37:55 +0000 UTC" deadline="2028-02-27 19:47:29.334546762 +0000 UTC"
Sep 12 00:42:56.653211 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:56.653207    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12811h4m32.68134282s"
Sep 12 00:42:56.718705 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:56.718686    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:56.819091 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:56.819042    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:56.919464 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:56.919442    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:56.946439 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:56.946421    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:42:57.020150 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:57.020124    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-121.ec2.internal\" not found"
Sep 12 00:42:57.064538 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.064514    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:42:57.069789 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.069741    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:42:57.087177 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.087003    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal"
Sep 12 00:42:57.122507 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.122410    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 12 00:42:57.123213 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.123192    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal"
Sep 12 00:42:57.149459 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.149438    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 12 00:42:57.462282 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.462205    2579 apiserver.go:52] "Watching apiserver"
Sep 12 00:42:57.470295 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.470270    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 12 00:42:57.471052 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.471026    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-ntsr9","kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z","openshift-dns/node-resolver-6q7h9","openshift-image-registry/node-ca-7m4gn","openshift-network-diagnostics/network-check-target-v2z26","kube-system/global-pull-secret-syncer-nggqt","openshift-cluster-node-tuning-operator/tuned-m7bsw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal","openshift-multus/multus-4nnlj","openshift-multus/multus-additional-cni-plugins-g4l4j","openshift-multus/network-metrics-daemon-k776z","openshift-network-operator/iptables-alerter-nmtfl","openshift-ovn-kubernetes/ovnkube-node-qzkgb"]
Sep 12 00:42:57.473655 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.473638    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:42:57.473746 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:57.473702    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:42:57.475904 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.475880    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.477997 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.477979    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:42:57.478098 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:57.478042    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:42:57.480195 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.480158    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.480325 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.480289    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 12 00:42:57.480612 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.480580    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 12 00:42:57.481013 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.480989    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 12 00:42:57.481140 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.481123    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xpnns\""
Sep 12 00:42:57.481224 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.481199    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 12 00:42:57.481327 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.481309    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 12 00:42:57.481444 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.481392    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 12 00:42:57.482413 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.482395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7m4gn"
Sep 12 00:42:57.483004 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.482982    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4lghz\""
Sep 12 00:42:57.484091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.484064    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 12 00:42:57.484811 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.484793    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 12 00:42:57.484915 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.484821    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:42:57.484915 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:57.484879    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:42:57.485371 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.485141    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 12 00:42:57.485371 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.485164    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 12 00:42:57.485371 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.485237    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 12 00:42:57.485575 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.485406    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 12 00:42:57.485575 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.485405    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 12 00:42:57.485575 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.485442    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 12 00:42:57.485575 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.485477    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nhqpq\""
Sep 12 00:42:57.486518 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.486501    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 12 00:42:57.487164 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.487144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ntsr9"
Sep 12 00:42:57.489362 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.489343    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 12 00:42:57.489646 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.489627    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 12 00:42:57.489726 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.489656    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.490068 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.490049    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jm9nw\""
Sep 12 00:42:57.491807 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.491778    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 12 00:42:57.492206 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.492193    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.492698 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.492678    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:42:57.492812 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.492796    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8rn8r\""
Sep 12 00:42:57.494861 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.494845    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 12 00:42:57.495068 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.495055    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rwqgw\""
Sep 12 00:42:57.495488 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.495462    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6q7h9"
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498022    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5-serviceca\") pod \"node-ca-7m4gn\" (UID: \"305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5\") " pod="openshift-image-registry/node-ca-7m4gn"
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498114    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gdphh\""
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8fd9311d-6922-42f5-8689-e768402de953-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498155    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-kubernetes\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-host\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-tmp\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjnx\" (UniqueName: \"kubernetes.io/projected/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-kube-api-access-xmjnx\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-run-systemd\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-run-ovn\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-cni-netd\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs\") pod \"network-metrics-daemon-k776z\" (UID: \"00498a5c-bd6a-4220-a017-53f01b8c1e92\") " pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-lib-modules\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-systemd-units\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498748    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 12 00:42:57.498916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498748    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx9x9\" (UniqueName: \"kubernetes.io/projected/00498a5c-bd6a-4220-a017-53f01b8c1e92-kube-api-access-tx9x9\") pod \"network-metrics-daemon-k776z\" (UID: \"00498a5c-bd6a-4220-a017-53f01b8c1e92\") " pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.498949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8fd9311d-6922-42f5-8689-e768402de953-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-modprobe-d\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-cni-bin\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45c04ace-43ff-47a5-a953-68957b92aba4-ovnkube-config\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499153    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499155    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5-host\") pod \"node-ca-7m4gn\" (UID: \"305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5\") " pod="openshift-image-registry/node-ca-7m4gn"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-kubelet-config\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-tuned\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-run-netns\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-log-socket\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45c04ace-43ff-47a5-a953-68957b92aba4-env-overrides\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45c04ace-43ff-47a5-a953-68957b92aba4-ovnkube-script-lib\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fd9311d-6922-42f5-8689-e768402de953-cnibin\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-systemd\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-run\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.499760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-node-log\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-run-ovn-kubernetes\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fd9311d-6922-42f5-8689-e768402de953-cni-binary-copy\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499743    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm5sk\" (UniqueName: \"kubernetes.io/projected/8fd9311d-6922-42f5-8689-e768402de953-kube-api-access-gm5sk\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fae396c5-d77f-4f6e-9881-b69beadbc497-konnectivity-ca\") pod \"konnectivity-agent-ntsr9\" (UID: \"fae396c5-d77f-4f6e-9881-b69beadbc497\") " pod="kube-system/konnectivity-agent-ntsr9"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-sys\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-var-lib-kubelet\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqw7n\" (UniqueName: \"kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n\") pod \"network-check-target-v2z26\" (UID: \"b4e3082f-9a39-472f-b400-48312f862ba1\") " pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-var-lib-openvswitch\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.499966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-etc-openvswitch\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.500045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-run-openvswitch\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.500096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7bgd\" (UniqueName: \"kubernetes.io/projected/45c04ace-43ff-47a5-a953-68957b92aba4-kube-api-access-d7bgd\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.500127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wc66\" (UniqueName: \"kubernetes.io/projected/305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5-kube-api-access-6wc66\") pod \"node-ca-7m4gn\" (UID: \"305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5\") " pod="openshift-image-registry/node-ca-7m4gn"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.500158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fd9311d-6922-42f5-8689-e768402de953-system-cni-dir\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.500191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fae396c5-d77f-4f6e-9881-b69beadbc497-agent-certs\") pod \"konnectivity-agent-ntsr9\" (UID: \"fae396c5-d77f-4f6e-9881-b69beadbc497\") " pod="kube-system/konnectivity-agent-ntsr9"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.500232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-slash\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.500460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.500261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45c04ace-43ff-47a5-a953-68957b92aba4-ovn-node-metrics-cert\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.501176 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.500295    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-dbus\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:42:57.501176 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.500326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fd9311d-6922-42f5-8689-e768402de953-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.501176 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.500379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-sysconfig\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.501176 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.500410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-sysctl-d\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.501176 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.500437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-sysctl-conf\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.501176 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.500528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-kubelet\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.501436 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.501382    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nmtfl"
Sep 12 00:42:57.502061 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.501809    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 12 00:42:57.502240 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.502223    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 12 00:42:57.502442 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.502422    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4z7gv\""
Sep 12 00:42:57.502529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.502512    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 12 00:42:57.503144 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.503117    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 12 00:42:57.504853 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.504474    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 12 00:42:57.504853 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.504512    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 12 00:42:57.504853 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.504742    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 12 00:42:57.505507 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.505220    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r88hb\""
Sep 12 00:42:57.588389 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.588360    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 12 00:42:57.589171 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.589124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal" event={"ID":"7cbe291fdf0e33f3b3d11babbfc8736a","Type":"ContainerStarted","Data":"7997e69c0e8c0d44e6ee97608af037036448c6aa2570d4e7bed368d394fdf46f"}
Sep 12 00:42:57.590328 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.590302    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal" event={"ID":"dd4b87aecbc86ab2cf93f2fecaacbc0b","Type":"ContainerStarted","Data":"69120cc37d3b04929912cb6e48c43106b93aa3f33f5a5c490a7366ca0889568f"}
Sep 12 00:42:57.598297 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.598274    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k84m9"
Sep 12 00:42:57.601408 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-var-lib-cni-multus\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.601490 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-kubelet\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.601490 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5-serviceca\") pod \"node-ca-7m4gn\" (UID: \"305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5\") " pod="openshift-image-registry/node-ca-7m4gn"
Sep 12 00:42:57.601490 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8fd9311d-6922-42f5-8689-e768402de953-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.601490 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-kubernetes\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.601714 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-kubelet\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.601714 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjnx\" (UniqueName: \"kubernetes.io/projected/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-kube-api-access-xmjnx\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.601714 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-kubernetes\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.601714 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-socket-dir\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.601714 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/45071600-05fe-42bf-af55-b93e22e162cc-iptables-alerter-script\") pod \"iptables-alerter-nmtfl\" (UID: \"45071600-05fe-42bf-af55-b93e22e162cc\") " pod="openshift-network-operator/iptables-alerter-nmtfl"
Sep 12 00:42:57.601714 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-run-ovn\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.601714 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs\") pod \"network-metrics-daemon-k776z\" (UID: \"00498a5c-bd6a-4220-a017-53f01b8c1e92\") " pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:42:57.601714 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601711    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-registration-dir\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggppp\" (UniqueName: \"kubernetes.io/projected/45071600-05fe-42bf-af55-b93e22e162cc-kube-api-access-ggppp\") pod \"iptables-alerter-nmtfl\" (UID: \"45071600-05fe-42bf-af55-b93e22e162cc\") " pod="openshift-network-operator/iptables-alerter-nmtfl"
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-run-ovn\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjpjt\" (UniqueName: \"kubernetes.io/projected/64c9aebc-9e60-472a-891a-7853806abd9e-kube-api-access-vjpjt\") pod \"node-resolver-6q7h9\" (UID: \"64c9aebc-9e60-472a-891a-7853806abd9e\") " pod="openshift-dns/node-resolver-6q7h9"
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-systemd-units\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-run-netns\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601874    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-systemd-units\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-multus-conf-dir\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:57.601916    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-sys-fs\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.601914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:57.601995    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs podName:00498a5c-bd6a-4220-a017-53f01b8c1e92 nodeName:}" failed. No retries permitted until 2026-09-12 00:42:58.101963285 +0000 UTC m=+3.021255065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs") pod "network-metrics-daemon-k776z" (UID: "00498a5c-bd6a-4220-a017-53f01b8c1e92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-cni-bin\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5-host\") pod \"node-ca-7m4gn\" (UID: \"305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5\") " pod="openshift-image-registry/node-ca-7m4gn"
Sep 12 00:42:57.602091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fd9311d-6922-42f5-8689-e768402de953-cnibin\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602086    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8fd9311d-6922-42f5-8689-e768402de953-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602096    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-cni-bin\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-systemd\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602141    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5-host\") pod \"node-ca-7m4gn\" (UID: \"305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5\") " pod="openshift-image-registry/node-ca-7m4gn"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-run\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-hostroot\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f6rz\" (UniqueName: \"kubernetes.io/projected/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-kube-api-access-6f6rz\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8fd9311d-6922-42f5-8689-e768402de953-cnibin\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-node-log\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-systemd\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-run-ovn-kubernetes\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-node-log\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-run\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fd9311d-6922-42f5-8689-e768402de953-cni-binary-copy\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602392    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-run-ovn-kubernetes\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602405    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5-serviceca\") pod \"node-ca-7m4gn\" (UID: \"305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5\") " pod="openshift-image-registry/node-ca-7m4gn"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-sys\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.602829 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-var-lib-kubelet\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-var-lib-kubelet\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2z55\" (UniqueName: \"kubernetes.io/projected/ea448e56-0b01-4890-a060-7a7de4a69b12-kube-api-access-v2z55\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-sys\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64c9aebc-9e60-472a-891a-7853806abd9e-hosts-file\") pod \"node-resolver-6q7h9\" (UID: \"64c9aebc-9e60-472a-891a-7853806abd9e\") " pod="openshift-dns/node-resolver-6q7h9"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-var-lib-openvswitch\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-run-openvswitch\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fd9311d-6922-42f5-8689-e768402de953-system-cni-dir\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-lib-modules\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-var-lib-openvswitch\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-var-lib-cni-bin\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-slash\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8fd9311d-6922-42f5-8689-e768402de953-system-cni-dir\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602723    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-run-openvswitch\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fd9311d-6922-42f5-8689-e768402de953-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602824    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-lib-modules\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-sysconfig\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.603719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-slash\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8fd9311d-6922-42f5-8689-e768402de953-cni-binary-copy\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ea448e56-0b01-4890-a060-7a7de4a69b12-multus-daemon-config\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-sysconfig\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-tmp\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8fd9311d-6922-42f5-8689-e768402de953-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-system-cni-dir\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.602991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-multus-cni-dir\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-run-systemd\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-run-systemd\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-cni-netd\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-cni-netd\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64c9aebc-9e60-472a-891a-7853806abd9e-tmp-dir\") pod \"node-resolver-6q7h9\" (UID: \"64c9aebc-9e60-472a-891a-7853806abd9e\") " pod="openshift-dns/node-resolver-6q7h9"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx9x9\" (UniqueName: \"kubernetes.io/projected/00498a5c-bd6a-4220-a017-53f01b8c1e92-kube-api-access-tx9x9\") pod \"network-metrics-daemon-k776z\" (UID: \"00498a5c-bd6a-4220-a017-53f01b8c1e92\") " pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8fd9311d-6922-42f5-8689-e768402de953-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-modprobe-d\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45c04ace-43ff-47a5-a953-68957b92aba4-ovnkube-config\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.604530 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-kubelet-config\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-tuned\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-run-multus-certs\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-etc-selinux\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-run-netns\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-kubelet-config\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-log-socket\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-host-run-netns\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45c04ace-43ff-47a5-a953-68957b92aba4-env-overrides\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-modprobe-d\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45c04ace-43ff-47a5-a953-68957b92aba4-ovnkube-script-lib\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603620    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-device-dir\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45071600-05fe-42bf-af55-b93e22e162cc-host-slash\") pod \"iptables-alerter-nmtfl\" (UID: \"45071600-05fe-42bf-af55-b93e22e162cc\") " pod="openshift-network-operator/iptables-alerter-nmtfl"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm5sk\" (UniqueName: \"kubernetes.io/projected/8fd9311d-6922-42f5-8689-e768402de953-kube-api-access-gm5sk\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fae396c5-d77f-4f6e-9881-b69beadbc497-konnectivity-ca\") pod \"konnectivity-agent-ntsr9\" (UID: \"fae396c5-d77f-4f6e-9881-b69beadbc497\") " pod="kube-system/konnectivity-agent-ntsr9"
Sep 12 00:42:57.605377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-run-k8s-cni-cncf-io\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.603994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-var-lib-kubelet\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqw7n\" (UniqueName: \"kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n\") pod \"network-check-target-v2z26\" (UID: \"b4e3082f-9a39-472f-b400-48312f862ba1\") " pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-etc-openvswitch\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7bgd\" (UniqueName: \"kubernetes.io/projected/45c04ace-43ff-47a5-a953-68957b92aba4-kube-api-access-d7bgd\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wc66\" (UniqueName: \"kubernetes.io/projected/305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5-kube-api-access-6wc66\") pod \"node-ca-7m4gn\" (UID: \"305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5\") " pod="openshift-image-registry/node-ca-7m4gn"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fae396c5-d77f-4f6e-9881-b69beadbc497-agent-certs\") pod \"konnectivity-agent-ntsr9\" (UID: \"fae396c5-d77f-4f6e-9881-b69beadbc497\") " pod="kube-system/konnectivity-agent-ntsr9"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-sysctl-conf\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea448e56-0b01-4890-a060-7a7de4a69b12-cni-binary-copy\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-etc-kubernetes\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45c04ace-43ff-47a5-a953-68957b92aba4-ovn-node-metrics-cert\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8fd9311d-6922-42f5-8689-e768402de953-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-dbus\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-sysctl-d\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-host\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-cnibin\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-multus-socket-dir-parent\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.606192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-sysctl-conf\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.607034 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604771    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-dbus\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:42:57.607034 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.604977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-log-socket\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.607034 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.605031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/45c04ace-43ff-47a5-a953-68957b92aba4-etc-openvswitch\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.607034 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.605074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-host\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.607034 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:57.605114    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:42:57.607034 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.605160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-sysctl-d\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.607034 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:57.605179    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret podName:c34751eb-27f9-4061-8e37-d49bd4bb8bd5 nodeName:}" failed. No retries permitted until 2026-09-12 00:42:58.105162799 +0000 UTC m=+3.024454568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret") pod "global-pull-secret-syncer-nggqt" (UID: "c34751eb-27f9-4061-8e37-d49bd4bb8bd5") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:42:57.607034 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.605543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/45c04ace-43ff-47a5-a953-68957b92aba4-ovnkube-script-lib\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.607034 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.605675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fae396c5-d77f-4f6e-9881-b69beadbc497-konnectivity-ca\") pod \"konnectivity-agent-ntsr9\" (UID: \"fae396c5-d77f-4f6e-9881-b69beadbc497\") " pod="kube-system/konnectivity-agent-ntsr9"
Sep 12 00:42:57.607034 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.606025    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/45c04ace-43ff-47a5-a953-68957b92aba4-env-overrides\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.607034 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.606124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/45c04ace-43ff-47a5-a953-68957b92aba4-ovnkube-config\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.607905 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.607885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-tmp\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.607905 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.607901    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-etc-tuned\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.608039 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.607930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/45c04ace-43ff-47a5-a953-68957b92aba4-ovn-node-metrics-cert\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.608111 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.608090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fae396c5-d77f-4f6e-9881-b69beadbc497-agent-certs\") pod \"konnectivity-agent-ntsr9\" (UID: \"fae396c5-d77f-4f6e-9881-b69beadbc497\") " pod="kube-system/konnectivity-agent-ntsr9"
Sep 12 00:42:57.608364 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.608349    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k84m9"
Sep 12 00:42:57.611902 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:57.611881    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:42:57.611996 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:57.611905    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:42:57.611996 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:57.611927    2579 projected.go:196] Error preparing data for projected volume kube-api-access-cqw7n for pod openshift-network-diagnostics/network-check-target-v2z26: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:42:57.611996 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:57.611978    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n podName:b4e3082f-9a39-472f-b400-48312f862ba1 nodeName:}" failed. No retries permitted until 2026-09-12 00:42:58.111964884 +0000 UTC m=+3.031256676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cqw7n" (UniqueName: "kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n") pod "network-check-target-v2z26" (UID: "b4e3082f-9a39-472f-b400-48312f862ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:42:57.615285 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.615261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm5sk\" (UniqueName: \"kubernetes.io/projected/8fd9311d-6922-42f5-8689-e768402de953-kube-api-access-gm5sk\") pod \"multus-additional-cni-plugins-g4l4j\" (UID: \"8fd9311d-6922-42f5-8689-e768402de953\") " pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.615956 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.615932    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wc66\" (UniqueName: \"kubernetes.io/projected/305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5-kube-api-access-6wc66\") pod \"node-ca-7m4gn\" (UID: \"305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5\") " pod="openshift-image-registry/node-ca-7m4gn"
Sep 12 00:42:57.617166 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.617150    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjnx\" (UniqueName: \"kubernetes.io/projected/b7ffa853-0a14-40d1-aa8b-28fe5d855e09-kube-api-access-xmjnx\") pod \"tuned-m7bsw\" (UID: \"b7ffa853-0a14-40d1-aa8b-28fe5d855e09\") " pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.617485 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.617459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7bgd\" (UniqueName: \"kubernetes.io/projected/45c04ace-43ff-47a5-a953-68957b92aba4-kube-api-access-d7bgd\") pod \"ovnkube-node-qzkgb\" (UID: \"45c04ace-43ff-47a5-a953-68957b92aba4\") " pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.617574 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.617542    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx9x9\" (UniqueName: \"kubernetes.io/projected/00498a5c-bd6a-4220-a017-53f01b8c1e92-kube-api-access-tx9x9\") pod \"network-metrics-daemon-k776z\" (UID: \"00498a5c-bd6a-4220-a017-53f01b8c1e92\") " pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:42:57.705648 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.705613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2z55\" (UniqueName: \"kubernetes.io/projected/ea448e56-0b01-4890-a060-7a7de4a69b12-kube-api-access-v2z55\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.705790 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.705744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64c9aebc-9e60-472a-891a-7853806abd9e-hosts-file\") pod \"node-resolver-6q7h9\" (UID: \"64c9aebc-9e60-472a-891a-7853806abd9e\") " pod="openshift-dns/node-resolver-6q7h9"
Sep 12 00:42:57.705790 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.705776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-var-lib-cni-bin\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.705923 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.705796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ea448e56-0b01-4890-a060-7a7de4a69b12-multus-daemon-config\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.705923 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.705857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-var-lib-cni-bin\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.705923 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.705861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64c9aebc-9e60-472a-891a-7853806abd9e-hosts-file\") pod \"node-resolver-6q7h9\" (UID: \"64c9aebc-9e60-472a-891a-7853806abd9e\") " pod="openshift-dns/node-resolver-6q7h9"
Sep 12 00:42:57.705923 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.705905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-system-cni-dir\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706120 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.705930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-multus-cni-dir\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706120 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.705958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64c9aebc-9e60-472a-891a-7853806abd9e-tmp-dir\") pod \"node-resolver-6q7h9\" (UID: \"64c9aebc-9e60-472a-891a-7853806abd9e\") " pod="openshift-dns/node-resolver-6q7h9"
Sep 12 00:42:57.706120 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.705983    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-system-cni-dir\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706120 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-run-multus-certs\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706120 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-etc-selinux\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.706120 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-device-dir\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.706120 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706077    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-run-multus-certs\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706120 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45071600-05fe-42bf-af55-b93e22e162cc-host-slash\") pod \"iptables-alerter-nmtfl\" (UID: \"45071600-05fe-42bf-af55-b93e22e162cc\") " pod="openshift-network-operator/iptables-alerter-nmtfl"
Sep 12 00:42:57.706120 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-run-k8s-cni-cncf-io\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-device-dir\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-multus-cni-dir\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-var-lib-kubelet\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/45071600-05fe-42bf-af55-b93e22e162cc-host-slash\") pod \"iptables-alerter-nmtfl\" (UID: \"45071600-05fe-42bf-af55-b93e22e162cc\") " pod="openshift-network-operator/iptables-alerter-nmtfl"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706161    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-run-k8s-cni-cncf-io\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706178    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-var-lib-kubelet\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea448e56-0b01-4890-a060-7a7de4a69b12-cni-binary-copy\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-etc-kubernetes\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-etc-selinux\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-cnibin\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-multus-socket-dir-parent\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-etc-kubernetes\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64c9aebc-9e60-472a-891a-7853806abd9e-tmp-dir\") pod \"node-resolver-6q7h9\" (UID: \"64c9aebc-9e60-472a-891a-7853806abd9e\") " pod="openshift-dns/node-resolver-6q7h9"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-var-lib-cni-multus\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706349    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-var-lib-cni-multus\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ea448e56-0b01-4890-a060-7a7de4a69b12-multus-daemon-config\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-socket-dir\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.706529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706374    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-multus-socket-dir-parent\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-cnibin\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/45071600-05fe-42bf-af55-b93e22e162cc-iptables-alerter-script\") pod \"iptables-alerter-nmtfl\" (UID: \"45071600-05fe-42bf-af55-b93e22e162cc\") " pod="openshift-network-operator/iptables-alerter-nmtfl"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-registration-dir\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggppp\" (UniqueName: \"kubernetes.io/projected/45071600-05fe-42bf-af55-b93e22e162cc-kube-api-access-ggppp\") pod \"iptables-alerter-nmtfl\" (UID: \"45071600-05fe-42bf-af55-b93e22e162cc\") " pod="openshift-network-operator/iptables-alerter-nmtfl"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706483    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-registration-dir\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjpjt\" (UniqueName: \"kubernetes.io/projected/64c9aebc-9e60-472a-891a-7853806abd9e-kube-api-access-vjpjt\") pod \"node-resolver-6q7h9\" (UID: \"64c9aebc-9e60-472a-891a-7853806abd9e\") " pod="openshift-dns/node-resolver-6q7h9"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-run-netns\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-multus-conf-dir\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-sys-fs\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-host-run-netns\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-hostroot\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-socket-dir\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-hostroot\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-sys-fs\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea448e56-0b01-4890-a060-7a7de4a69b12-multus-conf-dir\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.707283 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706686    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.707858 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f6rz\" (UniqueName: \"kubernetes.io/projected/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-kube-api-access-6f6rz\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.707858 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea448e56-0b01-4890-a060-7a7de4a69b12-cni-binary-copy\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.707858 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.706915    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/45071600-05fe-42bf-af55-b93e22e162cc-iptables-alerter-script\") pod \"iptables-alerter-nmtfl\" (UID: \"45071600-05fe-42bf-af55-b93e22e162cc\") " pod="openshift-network-operator/iptables-alerter-nmtfl"
Sep 12 00:42:57.723899 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.723830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2z55\" (UniqueName: \"kubernetes.io/projected/ea448e56-0b01-4890-a060-7a7de4a69b12-kube-api-access-v2z55\") pod \"multus-4nnlj\" (UID: \"ea448e56-0b01-4890-a060-7a7de4a69b12\") " pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.726087 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.726067    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjpjt\" (UniqueName: \"kubernetes.io/projected/64c9aebc-9e60-472a-891a-7853806abd9e-kube-api-access-vjpjt\") pod \"node-resolver-6q7h9\" (UID: \"64c9aebc-9e60-472a-891a-7853806abd9e\") " pod="openshift-dns/node-resolver-6q7h9"
Sep 12 00:42:57.726419 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.726397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggppp\" (UniqueName: \"kubernetes.io/projected/45071600-05fe-42bf-af55-b93e22e162cc-kube-api-access-ggppp\") pod \"iptables-alerter-nmtfl\" (UID: \"45071600-05fe-42bf-af55-b93e22e162cc\") " pod="openshift-network-operator/iptables-alerter-nmtfl"
Sep 12 00:42:57.728472 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.728446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f6rz\" (UniqueName: \"kubernetes.io/projected/d5b4c857-5e23-48bf-82c7-8fd0d34811a4-kube-api-access-6f6rz\") pod \"aws-ebs-csi-driver-node-lqr7z\" (UID: \"d5b4c857-5e23-48bf-82c7-8fd0d34811a4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.787088 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.787054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:42:57.794967 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.794943    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g4l4j"
Sep 12 00:42:57.804576 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.804555    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7m4gn"
Sep 12 00:42:57.810220 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.810203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ntsr9"
Sep 12 00:42:57.817210 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.817187    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m7bsw"
Sep 12 00:42:57.825884 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.825866    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4nnlj"
Sep 12 00:42:57.833550 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.833532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6q7h9"
Sep 12 00:42:57.840583 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.840566    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z"
Sep 12 00:42:57.847165 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:57.847148    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nmtfl"
Sep 12 00:42:58.110141 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.110073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:42:58.110271 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.110139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs\") pod \"network-metrics-daemon-k776z\" (UID: \"00498a5c-bd6a-4220-a017-53f01b8c1e92\") " pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:42:58.110271 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:58.110234    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:42:58.110271 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:58.110255    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:42:58.110408 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:58.110310    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs podName:00498a5c-bd6a-4220-a017-53f01b8c1e92 nodeName:}" failed. No retries permitted until 2026-09-12 00:42:59.110294744 +0000 UTC m=+4.029586531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs") pod "network-metrics-daemon-k776z" (UID: "00498a5c-bd6a-4220-a017-53f01b8c1e92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:42:58.110408 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:58.110326    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret podName:c34751eb-27f9-4061-8e37-d49bd4bb8bd5 nodeName:}" failed. No retries permitted until 2026-09-12 00:42:59.11031847 +0000 UTC m=+4.029610235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret") pod "global-pull-secret-syncer-nggqt" (UID: "c34751eb-27f9-4061-8e37-d49bd4bb8bd5") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:42:58.210833 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.210796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqw7n\" (UniqueName: \"kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n\") pod \"network-check-target-v2z26\" (UID: \"b4e3082f-9a39-472f-b400-48312f862ba1\") " pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:42:58.210987 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:58.210966    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:42:58.211044 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:58.210990    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:42:58.211044 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:58.211003    2579 projected.go:196] Error preparing data for projected volume kube-api-access-cqw7n for pod openshift-network-diagnostics/network-check-target-v2z26: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:42:58.211122 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:58.211055    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n podName:b4e3082f-9a39-472f-b400-48312f862ba1 nodeName:}" failed. No retries permitted until 2026-09-12 00:42:59.211041532 +0000 UTC m=+4.130333303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqw7n" (UniqueName: "kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n") pod "network-check-target-v2z26" (UID: "b4e3082f-9a39-472f-b400-48312f862ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:42:58.364225 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.364150    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 00:42:58.449780 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:58.449635    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c9aebc_9e60_472a_891a_7853806abd9e.slice/crio-f0a9178376f598b2d7bece31e6d86f66dc91c208d36ad2db4ba8707e6da8aaf9 WatchSource:0}: Error finding container f0a9178376f598b2d7bece31e6d86f66dc91c208d36ad2db4ba8707e6da8aaf9: Status 404 returned error can't find the container with id f0a9178376f598b2d7bece31e6d86f66dc91c208d36ad2db4ba8707e6da8aaf9
Sep 12 00:42:58.453460 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:58.452529    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305c7a25_c9a4_4f29_ae7a_ff9bf695f4b5.slice/crio-46feb6fca165b5ec9141d962513c3e6765a9ab09dbe8edfb162d4b79398134a6 WatchSource:0}: Error finding container 46feb6fca165b5ec9141d962513c3e6765a9ab09dbe8edfb162d4b79398134a6: Status 404 returned error can't find the container with id 46feb6fca165b5ec9141d962513c3e6765a9ab09dbe8edfb162d4b79398134a6
Sep 12 00:42:58.456644 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:58.456623    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45071600_05fe_42bf_af55_b93e22e162cc.slice/crio-cd4c46cb125f53f5ce49fd09947787c6d8e26040e2132e14d6f419117d95a16b WatchSource:0}: Error finding container cd4c46cb125f53f5ce49fd09947787c6d8e26040e2132e14d6f419117d95a16b: Status 404 returned error can't find the container with id cd4c46cb125f53f5ce49fd09947787c6d8e26040e2132e14d6f419117d95a16b
Sep 12 00:42:58.457537 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:58.457515    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ffa853_0a14_40d1_aa8b_28fe5d855e09.slice/crio-0814df4d7d270f030cd94a89d5402e19b3167341155d2b9b624c67cb871e7ae5 WatchSource:0}: Error finding container 0814df4d7d270f030cd94a89d5402e19b3167341155d2b9b624c67cb871e7ae5: Status 404 returned error can't find the container with id 0814df4d7d270f030cd94a89d5402e19b3167341155d2b9b624c67cb871e7ae5
Sep 12 00:42:58.459310 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:58.459257    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c04ace_43ff_47a5_a953_68957b92aba4.slice/crio-d78d2d77134ca68266b28739807b616e9bd79f9d36b0172adcc527079059526c WatchSource:0}: Error finding container d78d2d77134ca68266b28739807b616e9bd79f9d36b0172adcc527079059526c: Status 404 returned error can't find the container with id d78d2d77134ca68266b28739807b616e9bd79f9d36b0172adcc527079059526c
Sep 12 00:42:58.459908 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:58.459867    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae396c5_d77f_4f6e_9881_b69beadbc497.slice/crio-1bf8e970b77bc4d991a89c51f7d2f4a84488c2e1c2b288f615c0d1cc4a8d904a WatchSource:0}: Error finding container 1bf8e970b77bc4d991a89c51f7d2f4a84488c2e1c2b288f615c0d1cc4a8d904a: Status 404 returned error can't find the container with id 1bf8e970b77bc4d991a89c51f7d2f4a84488c2e1c2b288f615c0d1cc4a8d904a
Sep 12 00:42:58.461670 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:58.461637    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b4c857_5e23_48bf_82c7_8fd0d34811a4.slice/crio-df48ad7a23f77ea8315452c965e0a677fd8906428231a9bc36a4235464d8fb93 WatchSource:0}: Error finding container df48ad7a23f77ea8315452c965e0a677fd8906428231a9bc36a4235464d8fb93: Status 404 returned error can't find the container with id df48ad7a23f77ea8315452c965e0a677fd8906428231a9bc36a4235464d8fb93
Sep 12 00:42:58.462902 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:58.462884    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea448e56_0b01_4890_a060_7a7de4a69b12.slice/crio-18f54679d5ce3e46eae0d21df896eda94673ab8aee0821aecbabade8351d3dfe WatchSource:0}: Error finding container 18f54679d5ce3e46eae0d21df896eda94673ab8aee0821aecbabade8351d3dfe: Status 404 returned error can't find the container with id 18f54679d5ce3e46eae0d21df896eda94673ab8aee0821aecbabade8351d3dfe
Sep 12 00:42:58.464923 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:42:58.464902    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd9311d_6922_42f5_8689_e768402de953.slice/crio-37d6733955762b1cd551561b86afe4a90e7307f733cd88a84cb745af183ff960 WatchSource:0}: Error finding container 37d6733955762b1cd551561b86afe4a90e7307f733cd88a84cb745af183ff960: Status 404 returned error can't find the container with id 37d6733955762b1cd551561b86afe4a90e7307f733cd88a84cb745af183ff960
Sep 12 00:42:58.598148 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.598111    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z" event={"ID":"d5b4c857-5e23-48bf-82c7-8fd0d34811a4","Type":"ContainerStarted","Data":"df48ad7a23f77ea8315452c965e0a677fd8906428231a9bc36a4235464d8fb93"}
Sep 12 00:42:58.599157 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.599133    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ntsr9" event={"ID":"fae396c5-d77f-4f6e-9881-b69beadbc497","Type":"ContainerStarted","Data":"1bf8e970b77bc4d991a89c51f7d2f4a84488c2e1c2b288f615c0d1cc4a8d904a"}
Sep 12 00:42:58.601444 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.601422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nmtfl" event={"ID":"45071600-05fe-42bf-af55-b93e22e162cc","Type":"ContainerStarted","Data":"cd4c46cb125f53f5ce49fd09947787c6d8e26040e2132e14d6f419117d95a16b"}
Sep 12 00:42:58.603214 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.603194    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m7bsw" event={"ID":"b7ffa853-0a14-40d1-aa8b-28fe5d855e09","Type":"ContainerStarted","Data":"0814df4d7d270f030cd94a89d5402e19b3167341155d2b9b624c67cb871e7ae5"}
Sep 12 00:42:58.604288 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.604269    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7m4gn" event={"ID":"305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5","Type":"ContainerStarted","Data":"46feb6fca165b5ec9141d962513c3e6765a9ab09dbe8edfb162d4b79398134a6"}
Sep 12 00:42:58.605985 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.605960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal" event={"ID":"7cbe291fdf0e33f3b3d11babbfc8736a","Type":"ContainerStarted","Data":"693a2f1406c4047c118cf39660084ccdae255890410e61043939da8c00f98c31"}
Sep 12 00:42:58.611109 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.611092    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4l4j" event={"ID":"8fd9311d-6922-42f5-8689-e768402de953","Type":"ContainerStarted","Data":"37d6733955762b1cd551561b86afe4a90e7307f733cd88a84cb745af183ff960"}
Sep 12 00:42:58.612262 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.612211    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 00:37:57 +0000 UTC" deadline="2028-02-15 16:53:46.433490275 +0000 UTC"
Sep 12 00:42:58.612262 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.612260    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12520h10m47.821235046s"
Sep 12 00:42:58.612780 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.612760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb" event={"ID":"45c04ace-43ff-47a5-a953-68957b92aba4","Type":"ContainerStarted","Data":"d78d2d77134ca68266b28739807b616e9bd79f9d36b0172adcc527079059526c"}
Sep 12 00:42:58.613944 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.613920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6q7h9" event={"ID":"64c9aebc-9e60-472a-891a-7853806abd9e","Type":"ContainerStarted","Data":"f0a9178376f598b2d7bece31e6d86f66dc91c208d36ad2db4ba8707e6da8aaf9"}
Sep 12 00:42:58.617190 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.617140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4nnlj" event={"ID":"ea448e56-0b01-4890-a060-7a7de4a69b12","Type":"ContainerStarted","Data":"18f54679d5ce3e46eae0d21df896eda94673ab8aee0821aecbabade8351d3dfe"}
Sep 12 00:42:58.621256 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:58.621213    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-121.ec2.internal" podStartSLOduration=1.62120286 podStartE2EDuration="1.62120286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:42:58.621169106 +0000 UTC m=+3.540460918" watchObservedRunningTime="2026-09-12 00:42:58.62120286 +0000 UTC m=+3.540494647"
Sep 12 00:42:59.119052 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:59.118312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:42:59.119052 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:59.118392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs\") pod \"network-metrics-daemon-k776z\" (UID: \"00498a5c-bd6a-4220-a017-53f01b8c1e92\") " pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:42:59.119052 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:59.118520    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:42:59.119052 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:59.118579    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs podName:00498a5c-bd6a-4220-a017-53f01b8c1e92 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:01.118561715 +0000 UTC m=+6.037853494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs") pod "network-metrics-daemon-k776z" (UID: "00498a5c-bd6a-4220-a017-53f01b8c1e92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:42:59.119052 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:59.118969    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:42:59.119052 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:59.119017    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret podName:c34751eb-27f9-4061-8e37-d49bd4bb8bd5 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:01.119000811 +0000 UTC m=+6.038292581 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret") pod "global-pull-secret-syncer-nggqt" (UID: "c34751eb-27f9-4061-8e37-d49bd4bb8bd5") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:42:59.219543 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:59.219505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqw7n\" (UniqueName: \"kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n\") pod \"network-check-target-v2z26\" (UID: \"b4e3082f-9a39-472f-b400-48312f862ba1\") " pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:42:59.219736 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:59.219720    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:42:59.219798 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:59.219744    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:42:59.219798 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:59.219756    2579 projected.go:196] Error preparing data for projected volume kube-api-access-cqw7n for pod openshift-network-diagnostics/network-check-target-v2z26: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:42:59.219900 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:59.219826    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n podName:b4e3082f-9a39-472f-b400-48312f862ba1 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:01.219808677 +0000 UTC m=+6.139100447 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqw7n" (UniqueName: "kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n") pod "network-check-target-v2z26" (UID: "b4e3082f-9a39-472f-b400-48312f862ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:42:59.587779 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:59.587712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:42:59.587779 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:59.587734    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:42:59.588615 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:59.587850    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:42:59.588615 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:59.588030    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:42:59.588615 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:59.588239    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:42:59.588615 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:42:59.588328    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:42:59.612765 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:59.612732    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 00:37:57 +0000 UTC" deadline="2028-02-18 10:20:31.632607409 +0000 UTC"
Sep 12 00:42:59.612765 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:59.612764    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12585h37m32.019847454s"
Sep 12 00:42:59.629305 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:59.628342    2579 generic.go:356] "Generic (PLEG): container finished" podID="dd4b87aecbc86ab2cf93f2fecaacbc0b" containerID="2aabe2ecbd17bf49890a8788e046099d91e6887ff528aeb67559948e4e763cb3" exitCode=0
Sep 12 00:42:59.629305 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:42:59.629253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal" event={"ID":"dd4b87aecbc86ab2cf93f2fecaacbc0b","Type":"ContainerDied","Data":"2aabe2ecbd17bf49890a8788e046099d91e6887ff528aeb67559948e4e763cb3"}
Sep 12 00:43:00.637964 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:00.637856    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal" event={"ID":"dd4b87aecbc86ab2cf93f2fecaacbc0b","Type":"ContainerStarted","Data":"5391121e4187f2aaf53a23c77a8ac98c22ef476e45471256d42deefa24b2b23e"}
Sep 12 00:43:01.137162 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:01.137075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:01.137162 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:01.137160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs\") pod \"network-metrics-daemon-k776z\" (UID: \"00498a5c-bd6a-4220-a017-53f01b8c1e92\") " pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:01.137380 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:01.137294    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:01.137380 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:01.137354    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs podName:00498a5c-bd6a-4220-a017-53f01b8c1e92 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:05.137335241 +0000 UTC m=+10.056627009 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs") pod "network-metrics-daemon-k776z" (UID: "00498a5c-bd6a-4220-a017-53f01b8c1e92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:01.137863 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:01.137838    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:01.137969 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:01.137896    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret podName:c34751eb-27f9-4061-8e37-d49bd4bb8bd5 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:05.137880591 +0000 UTC m=+10.057172360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret") pod "global-pull-secret-syncer-nggqt" (UID: "c34751eb-27f9-4061-8e37-d49bd4bb8bd5") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:01.237704 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:01.237667    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqw7n\" (UniqueName: \"kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n\") pod \"network-check-target-v2z26\" (UID: \"b4e3082f-9a39-472f-b400-48312f862ba1\") " pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:01.237876 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:01.237854    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:43:01.237938 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:01.237880    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:43:01.237938 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:01.237893    2579 projected.go:196] Error preparing data for projected volume kube-api-access-cqw7n for pod openshift-network-diagnostics/network-check-target-v2z26: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:01.238041 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:01.237949    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n podName:b4e3082f-9a39-472f-b400-48312f862ba1 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:05.237931806 +0000 UTC m=+10.157223577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqw7n" (UniqueName: "kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n") pod "network-check-target-v2z26" (UID: "b4e3082f-9a39-472f-b400-48312f862ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:01.585911 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:01.585781    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:01.585911 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:01.585795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:01.585911 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:01.585895    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:43:01.586163 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:01.585979    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:43:01.586163 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:01.586044    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:01.586311 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:01.586150    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:43:03.586172 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:03.585696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:03.586172 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:03.585729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:03.586172 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:03.585696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:03.586172 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:03.585821    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:43:03.586172 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:03.585895    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:43:03.586172 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:03.585968    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:43:05.169985 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:05.169798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:05.169985 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:05.169859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs\") pod \"network-metrics-daemon-k776z\" (UID: \"00498a5c-bd6a-4220-a017-53f01b8c1e92\") " pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:05.169985 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:05.169951    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:05.170560 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:05.170012    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs podName:00498a5c-bd6a-4220-a017-53f01b8c1e92 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:13.169993496 +0000 UTC m=+18.089285261 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs") pod "network-metrics-daemon-k776z" (UID: "00498a5c-bd6a-4220-a017-53f01b8c1e92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:05.170560 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:05.169951    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:05.170560 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:05.170391    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret podName:c34751eb-27f9-4061-8e37-d49bd4bb8bd5 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:13.170370387 +0000 UTC m=+18.089662154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret") pod "global-pull-secret-syncer-nggqt" (UID: "c34751eb-27f9-4061-8e37-d49bd4bb8bd5") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:05.270799 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:05.270763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqw7n\" (UniqueName: \"kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n\") pod \"network-check-target-v2z26\" (UID: \"b4e3082f-9a39-472f-b400-48312f862ba1\") " pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:05.270964 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:05.270901    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:43:05.270964 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:05.270918    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:43:05.270964 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:05.270927    2579 projected.go:196] Error preparing data for projected volume kube-api-access-cqw7n for pod openshift-network-diagnostics/network-check-target-v2z26: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:05.271120 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:05.270979    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n podName:b4e3082f-9a39-472f-b400-48312f862ba1 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:13.270962769 +0000 UTC m=+18.190254536 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqw7n" (UniqueName: "kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n") pod "network-check-target-v2z26" (UID: "b4e3082f-9a39-472f-b400-48312f862ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:05.585874 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:05.585795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:05.586025 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:05.585880    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:43:05.586025 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:05.585972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:05.586149 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:05.586071    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:43:05.586149 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:05.586118    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:05.586219 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:05.586176    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:43:07.584895 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:07.584858    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:07.585301 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:07.584863    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:07.585301 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:07.585008    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:43:07.585301 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:07.585072    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:43:07.585301 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:07.584868    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:07.585301 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:07.585165    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:43:09.585021 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:09.584983    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:09.585021 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:09.584984    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:09.585518 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:09.585119    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:43:09.585518 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:09.585146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:09.585518 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:09.585268    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:43:09.585518 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:09.585366    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:43:11.584950 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:11.584920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:11.585394 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:11.584968    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:11.585394 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:11.584920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:11.585394 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:11.585040    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:43:11.585394 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:11.585176    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:43:11.585394 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:11.585271    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:43:13.227787 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:13.227755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:13.228230 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:13.227812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs\") pod \"network-metrics-daemon-k776z\" (UID: \"00498a5c-bd6a-4220-a017-53f01b8c1e92\") " pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:13.228230 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:13.227959    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:13.228230 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:13.227971    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:13.228230 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:13.228032    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret podName:c34751eb-27f9-4061-8e37-d49bd4bb8bd5 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:29.228009267 +0000 UTC m=+34.147301047 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret") pod "global-pull-secret-syncer-nggqt" (UID: "c34751eb-27f9-4061-8e37-d49bd4bb8bd5") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:13.228230 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:13.228049    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs podName:00498a5c-bd6a-4220-a017-53f01b8c1e92 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:29.228042021 +0000 UTC m=+34.147333785 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs") pod "network-metrics-daemon-k776z" (UID: "00498a5c-bd6a-4220-a017-53f01b8c1e92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:13.328874 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:13.328846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqw7n\" (UniqueName: \"kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n\") pod \"network-check-target-v2z26\" (UID: \"b4e3082f-9a39-472f-b400-48312f862ba1\") " pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:13.329063 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:13.329046    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:43:13.329120 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:13.329070    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:43:13.329120 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:13.329079    2579 projected.go:196] Error preparing data for projected volume kube-api-access-cqw7n for pod openshift-network-diagnostics/network-check-target-v2z26: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:13.329199 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:13.329131    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n podName:b4e3082f-9a39-472f-b400-48312f862ba1 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:29.329113185 +0000 UTC m=+34.248404954 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqw7n" (UniqueName: "kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n") pod "network-check-target-v2z26" (UID: "b4e3082f-9a39-472f-b400-48312f862ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:13.584911 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:13.584831    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:13.584911 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:13.584862    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:13.585117 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:13.584831    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:13.585117 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:13.584958    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:43:13.585117 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:13.585038    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:43:13.585246 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:13.585126    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:43:15.586262 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:15.586221    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:15.586666 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:15.586323    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:15.586666 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:15.586368    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:43:15.586666 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:15.586409    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:43:15.586666 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:15.586434    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:15.586666 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:15.586501    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:43:16.675821 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.675631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z" event={"ID":"d5b4c857-5e23-48bf-82c7-8fd0d34811a4","Type":"ContainerStarted","Data":"7692a364897e7bfa937b9d374be26dade8a1a39f8d50e215e461ae2935cc37e9"}
Sep 12 00:43:16.677009 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.676984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ntsr9" event={"ID":"fae396c5-d77f-4f6e-9881-b69beadbc497","Type":"ContainerStarted","Data":"9322f5289994034cc2b7d4505521503903921979530d9734534f02835d6735e5"}
Sep 12 00:43:16.678545 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.678523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m7bsw" event={"ID":"b7ffa853-0a14-40d1-aa8b-28fe5d855e09","Type":"ContainerStarted","Data":"b11667b3564a72e743b9ddd11957208170506d5c525636ab3987952fc70922ed"}
Sep 12 00:43:16.680309 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.680276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7m4gn" event={"ID":"305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5","Type":"ContainerStarted","Data":"20fb3c5deefc433a883b6fdf92f7ee55201cabd1bedab737d3f89c549bff8897"}
Sep 12 00:43:16.681948 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.681928    2579 generic.go:356] "Generic (PLEG): container finished" podID="8fd9311d-6922-42f5-8689-e768402de953" containerID="a1f6eb89772e71ba7d349df9a364e02a3209da8802b488916707a5575ff17e7c" exitCode=0
Sep 12 00:43:16.682056 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.681981    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4l4j" event={"ID":"8fd9311d-6922-42f5-8689-e768402de953","Type":"ContainerDied","Data":"a1f6eb89772e71ba7d349df9a364e02a3209da8802b488916707a5575ff17e7c"}
Sep 12 00:43:16.685314 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.685296    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 00:43:16.685719 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.685694    2579 generic.go:356] "Generic (PLEG): container finished" podID="45c04ace-43ff-47a5-a953-68957b92aba4" containerID="efc6256e37aeef40e98195b54161956e5efaeccc84ded2be7a9c3cf36530d116" exitCode=1
Sep 12 00:43:16.685811 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.685743    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb" event={"ID":"45c04ace-43ff-47a5-a953-68957b92aba4","Type":"ContainerStarted","Data":"c0f4683cfd0a5b2d62d475bbdc60c4532a084bbcda4ddb267ef9ef4e52102517"}
Sep 12 00:43:16.685867 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.685833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb" event={"ID":"45c04ace-43ff-47a5-a953-68957b92aba4","Type":"ContainerStarted","Data":"2a0c6651d338cebcbd9ed41427f5f709248d30215df029508ac3a141aa6363a0"}
Sep 12 00:43:16.685914 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.685901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb" event={"ID":"45c04ace-43ff-47a5-a953-68957b92aba4","Type":"ContainerStarted","Data":"1c882bab2e31596a39d0b79e15ed96e351ba08233e15e9c4b4f391de5718368e"}
Sep 12 00:43:16.686016 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.686001    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb" event={"ID":"45c04ace-43ff-47a5-a953-68957b92aba4","Type":"ContainerStarted","Data":"360f8fec67816c80d9dda7f8e3d3c75a7491cee2891a6e2812efe6feb517fe69"}
Sep 12 00:43:16.686101 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.686045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb" event={"ID":"45c04ace-43ff-47a5-a953-68957b92aba4","Type":"ContainerDied","Data":"efc6256e37aeef40e98195b54161956e5efaeccc84ded2be7a9c3cf36530d116"}
Sep 12 00:43:16.686101 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.686077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb" event={"ID":"45c04ace-43ff-47a5-a953-68957b92aba4","Type":"ContainerStarted","Data":"23edcb5c547cdcb14451f2b7773d82478bef27bf2e2f18ff80929b5f512cfc73"}
Sep 12 00:43:16.687817 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.687695    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6q7h9" event={"ID":"64c9aebc-9e60-472a-891a-7853806abd9e","Type":"ContainerStarted","Data":"9f50cc02dcef1ecc34dfcea783bb32ef566e99b331da95ab7985d317a1314af4"}
Sep 12 00:43:16.689377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.689357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4nnlj" event={"ID":"ea448e56-0b01-4890-a060-7a7de4a69b12","Type":"ContainerStarted","Data":"e8b7f270d3185191ac01aceac22ddf7a887ca56595addf5c92895585bdfc3a76"}
Sep 12 00:43:16.694931 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.694896    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-121.ec2.internal" podStartSLOduration=19.694883427 podStartE2EDuration="19.694883427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:00.681052915 +0000 UTC m=+5.600344701" watchObservedRunningTime="2026-09-12 00:43:16.694883427 +0000 UTC m=+21.614175227"
Sep 12 00:43:16.695330 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.695301    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ntsr9" podStartSLOduration=3.233713675 podStartE2EDuration="20.695293717s" totalImagesPullingTime="17.461580042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:16.694780272 +0000 UTC m=+21.614072061" watchObservedRunningTime="2026-09-12 00:43:16.695293717 +0000 UTC m=+21.614585504"
Sep 12 00:43:16.716774 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.716730    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m7bsw" podStartSLOduration=3.207810329 podStartE2EDuration="20.716716166s" totalImagesPullingTime="17.508905837s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:16.716203783 +0000 UTC m=+21.635495571" watchObservedRunningTime="2026-09-12 00:43:16.716716166 +0000 UTC m=+21.636007952"
Sep 12 00:43:16.734726 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.734679    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6q7h9" podStartSLOduration=3.218795791 podStartE2EDuration="20.734660873s" totalImagesPullingTime="17.515865082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:16.73414156 +0000 UTC m=+21.653433348" watchObservedRunningTime="2026-09-12 00:43:16.734660873 +0000 UTC m=+21.653952664"
Sep 12 00:43:16.827841 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.827664    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7m4gn" podStartSLOduration=4.359801119 podStartE2EDuration="21.827649545s" totalImagesPullingTime="17.467848426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:16.826855472 +0000 UTC m=+21.746147258" watchObservedRunningTime="2026-09-12 00:43:16.827649545 +0000 UTC m=+21.746941332"
Sep 12 00:43:16.928220 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:16.926865    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4nnlj" podStartSLOduration=3.01573945 podStartE2EDuration="20.926846675s" totalImagesPullingTime="17.911107225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:16.926763735 +0000 UTC m=+21.846055515" watchObservedRunningTime="2026-09-12 00:43:16.926846675 +0000 UTC m=+21.846138463"
Sep 12 00:43:17.585691 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:17.585657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:17.585814 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:17.585689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:17.585814 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:17.585657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:17.585814 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:17.585786    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:43:17.585928 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:17.585872    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:43:17.585975 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:17.585952    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:43:17.645357 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:17.645292    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 12 00:43:17.693791 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:17.693732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z" event={"ID":"d5b4c857-5e23-48bf-82c7-8fd0d34811a4","Type":"ContainerStarted","Data":"5280426e706ebc9c71a6fcce9ce27685516d7e4a6aeb97642bb1282e1113fed0"}
Sep 12 00:43:17.694590 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:17.694506    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-12T00:43:17.645308392Z","UUID":"85259aa0-7229-47e5-8447-123116eb20fb","Handler":null,"Name":"","Endpoint":""}
Sep 12 00:43:17.695222 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:17.695198    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nmtfl" event={"ID":"45071600-05fe-42bf-af55-b93e22e162cc","Type":"ContainerStarted","Data":"94921e0881f43287c9e969a3763de60fd9e7510a7458bac1de403cb8e055bd6d"}
Sep 12 00:43:17.696171 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:17.696071    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 12 00:43:17.696171 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:17.696101    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 12 00:43:17.710973 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:17.710934    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nmtfl" podStartSLOduration=4.202640436 podStartE2EDuration="21.710920941s" totalImagesPullingTime="17.508280505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:17.71051685 +0000 UTC m=+22.629808636" watchObservedRunningTime="2026-09-12 00:43:17.710920941 +0000 UTC m=+22.630212727"
Sep 12 00:43:19.585445 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:19.585415    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:19.585920 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:19.585420    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:19.585920 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:19.585514    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:43:19.585920 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:19.585544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:19.585920 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:19.585653    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:43:19.585920 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:19.585758    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:43:19.702374 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:19.702339    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z" event={"ID":"d5b4c857-5e23-48bf-82c7-8fd0d34811a4","Type":"ContainerStarted","Data":"0f5ee2d3e903dbc34b0029ab410c9061ca25a41738edbb170e9ea79845fd403b"}
Sep 12 00:43:19.706194 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:19.706173    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 00:43:19.706643 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:19.706619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb" event={"ID":"45c04ace-43ff-47a5-a953-68957b92aba4","Type":"ContainerStarted","Data":"26a41371a25201289a437d74daa0beb46bd8afd83e8b47561e3b5d2b03db07ff"}
Sep 12 00:43:19.723905 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:19.723856    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqr7z" podStartSLOduration=3.656482927 podStartE2EDuration="23.723835869s" totalImagesPullingTime="20.067352942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:42:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:19.723019341 +0000 UTC m=+24.642311129" watchObservedRunningTime="2026-09-12 00:43:19.723835869 +0000 UTC m=+24.643127658"
Sep 12 00:43:19.962232 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:19.962147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ntsr9"
Sep 12 00:43:19.963266 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:19.963244    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ntsr9"
Sep 12 00:43:21.585536 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:21.585307    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:21.586136 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:21.585344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:21.586136 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:21.585670    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:43:21.586136 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:21.585375    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:21.586136 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:21.585702    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:43:21.586136 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:21.585779    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:43:21.712976 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:21.712951    2579 generic.go:356] "Generic (PLEG): container finished" podID="8fd9311d-6922-42f5-8689-e768402de953" containerID="e4ee26f8ae2ff871cdb255624b1a908e2b6b209257f5a16a1137f6471f578436" exitCode=0
Sep 12 00:43:21.713080 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:21.713014    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4l4j" event={"ID":"8fd9311d-6922-42f5-8689-e768402de953","Type":"ContainerDied","Data":"e4ee26f8ae2ff871cdb255624b1a908e2b6b209257f5a16a1137f6471f578436"}
Sep 12 00:43:21.716649 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:21.716629    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 00:43:21.716958 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:21.716939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb" event={"ID":"45c04ace-43ff-47a5-a953-68957b92aba4","Type":"ContainerStarted","Data":"2bcc99bcbd0962ef425494b1156928c3682923c5ce08910c57c7cd57d7508439"}
Sep 12 00:43:21.717230 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:21.717199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:43:21.717308 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:21.717295    2579 scope.go:117] "RemoveContainer" containerID="efc6256e37aeef40e98195b54161956e5efaeccc84ded2be7a9c3cf36530d116"
Sep 12 00:43:21.731686 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:21.731668    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:43:22.721313 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:22.721223    2579 generic.go:356] "Generic (PLEG): container finished" podID="8fd9311d-6922-42f5-8689-e768402de953" containerID="3df865a6aaa7d7d0c0e2f5fd60a9418cba8d0e0926bea93d2b581216662c912e" exitCode=0
Sep 12 00:43:22.721737 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:22.721300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4l4j" event={"ID":"8fd9311d-6922-42f5-8689-e768402de953","Type":"ContainerDied","Data":"3df865a6aaa7d7d0c0e2f5fd60a9418cba8d0e0926bea93d2b581216662c912e"}
Sep 12 00:43:22.725581 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:22.725563    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 00:43:22.725976 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:22.725955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb" event={"ID":"45c04ace-43ff-47a5-a953-68957b92aba4","Type":"ContainerStarted","Data":"4620eb0a1a221693c41f1ede5799d4522a562636d9266938c8e70e11007d27e2"}
Sep 12 00:43:22.726312 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:22.726292    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:43:22.726426 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:22.726361    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:43:22.741188 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:22.741168    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:43:22.768589 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:22.768553    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb" podStartSLOduration=10.043595024 podStartE2EDuration="27.76854068s" totalImagesPullingTime="17.724945656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:22.7672477 +0000 UTC m=+27.686539485" watchObservedRunningTime="2026-09-12 00:43:22.76854068 +0000 UTC m=+27.687832636"
Sep 12 00:43:23.198077 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:23.197994    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v2z26"]
Sep 12 00:43:23.198259 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:23.198133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:23.198259 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:23.198241    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:43:23.202966 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:23.202939    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nggqt"]
Sep 12 00:43:23.203143 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:23.203046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:23.203211 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:23.203143    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:43:23.203734 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:23.203708    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k776z"]
Sep 12 00:43:23.203819 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:23.203806    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:23.204023 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:23.203899    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:43:23.730050 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:23.730016    2579 generic.go:356] "Generic (PLEG): container finished" podID="8fd9311d-6922-42f5-8689-e768402de953" containerID="8e379aa422c2af3f3b1be31bb78350f34b32a587861e5ec214d9fe30fa114f46" exitCode=0
Sep 12 00:43:23.730483 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:23.730097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4l4j" event={"ID":"8fd9311d-6922-42f5-8689-e768402de953","Type":"ContainerDied","Data":"8e379aa422c2af3f3b1be31bb78350f34b32a587861e5ec214d9fe30fa114f46"}
Sep 12 00:43:23.772613 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:23.772580    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ntsr9"
Sep 12 00:43:23.772820 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:23.772670    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 12 00:43:23.773293 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:23.773274    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ntsr9"
Sep 12 00:43:24.585231 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:24.585203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:24.585383 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:24.585203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:24.585383 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:24.585291    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:43:24.585383 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:24.585360    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:43:25.586968 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:25.586818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:25.586968 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:25.586945    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:43:26.585513 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:26.585481    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:26.585692 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:26.585482    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:26.585692 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:26.585629    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:43:26.585692 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:26.585672    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:43:27.584898 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:27.584823    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:27.585306 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:27.584969    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k776z" podUID="00498a5c-bd6a-4220-a017-53f01b8c1e92"
Sep 12 00:43:28.584934 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.584737    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:28.585325 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.584735    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:28.585325 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:28.584985    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nggqt" podUID="c34751eb-27f9-4061-8e37-d49bd4bb8bd5"
Sep 12 00:43:28.585325 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:28.585043    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v2z26" podUID="b4e3082f-9a39-472f-b400-48312f862ba1"
Sep 12 00:43:28.744393 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.744360    2579 generic.go:356] "Generic (PLEG): container finished" podID="8fd9311d-6922-42f5-8689-e768402de953" containerID="56a6fcf2bc8c327f60c3a20319ad3122b85e21a14903f19644d2c804669733f6" exitCode=0
Sep 12 00:43:28.744517 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.744419    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4l4j" event={"ID":"8fd9311d-6922-42f5-8689-e768402de953","Type":"ContainerDied","Data":"56a6fcf2bc8c327f60c3a20319ad3122b85e21a14903f19644d2c804669733f6"}
Sep 12 00:43:28.827329 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.827307    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-121.ec2.internal" event="NodeReady"
Sep 12 00:43:28.827437 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.827427    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 12 00:43:28.865354 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.865333    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7957768c6f-z2v8k"]
Sep 12 00:43:28.874414 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.874394    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9b45fb8fb-q97p8"]
Sep 12 00:43:28.874549 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.874527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:28.878398 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.878378    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rszm7\""
Sep 12 00:43:28.878398 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.878387    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 12 00:43:28.878557 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.878381    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 12 00:43:28.883368 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.883349    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 12 00:43:28.889621 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.888779    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"]
Sep 12 00:43:28.889780 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.889756    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 12 00:43:28.889855 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.889786    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9b45fb8fb-q97p8"
Sep 12 00:43:28.894070 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.892050    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 12 00:43:28.894070 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.893378    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-qzvc5\""
Sep 12 00:43:28.894070 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.893471    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 12 00:43:28.894070 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.893521    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 12 00:43:28.898682 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.894873    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 12 00:43:28.910851 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.910829    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"]
Sep 12 00:43:28.910990 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.910972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:28.913744 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.913542    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 12 00:43:28.914439 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.914421    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 12 00:43:28.914946 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.914927    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 12 00:43:28.915040 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.914946    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 12 00:43:28.928723 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.928705    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"]
Sep 12 00:43:28.928807 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.928737    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7957768c6f-z2v8k"]
Sep 12 00:43:28.928807 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.928760    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9b45fb8fb-q97p8"]
Sep 12 00:43:28.928807 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.928774    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"]
Sep 12 00:43:28.928807 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.928795    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sbqdk"]
Sep 12 00:43:28.929003 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.928817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"
Sep 12 00:43:28.931405 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.931388    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 12 00:43:28.950731 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.950714    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xbw45"]
Sep 12 00:43:28.950828 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.950819    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:28.953405 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.953389    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 12 00:43:28.953585 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.953570    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 12 00:43:28.953686 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.953577    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2gxdg\""
Sep 12 00:43:28.969014 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.968994    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xbw45"]
Sep 12 00:43:28.969093 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.969019    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sbqdk"]
Sep 12 00:43:28.969093 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.969089    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xbw45"
Sep 12 00:43:28.971456 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.971434    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qtwt4\""
Sep 12 00:43:28.971662 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.971650    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 12 00:43:28.971875 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.971860    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 12 00:43:28.972282 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:28.972268    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 12 00:43:29.041306 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.041277    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ntsr9_fae396c5-d77f-4f6e-9881-b69beadbc497/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:29.044216 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e3478ef0-9b7d-4705-b38d-7fdb72c084ad-klusterlet-config\") pod \"klusterlet-addon-workmgr-7445d698fb-tjvm6\" (UID: \"e3478ef0-9b7d-4705-b38d-7fdb72c084ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"
Sep 12 00:43:29.044278 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.044278 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:29.044278 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/08d25537-fb6c-489f-83a9-aa3b0f73384c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-9b45fb8fb-q97p8\" (UID: \"08d25537-fb6c-489f-83a9-aa3b0f73384c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9b45fb8fb-q97p8"
Sep 12 00:43:29.044437 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlfgf\" (UniqueName: \"kubernetes.io/projected/e3478ef0-9b7d-4705-b38d-7fdb72c084ad-kube-api-access-wlfgf\") pod \"klusterlet-addon-workmgr-7445d698fb-tjvm6\" (UID: \"e3478ef0-9b7d-4705-b38d-7fdb72c084ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"
Sep 12 00:43:29.044437 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-tmp-dir\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:29.044437 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-hub\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.044437 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894hf\" (UniqueName: \"kubernetes.io/projected/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-kube-api-access-894hf\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:29.044437 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77hxf\" (UniqueName: \"kubernetes.io/projected/08d25537-fb6c-489f-83a9-aa3b0f73384c-kube-api-access-77hxf\") pod \"managed-serviceaccount-addon-agent-9b45fb8fb-q97p8\" (UID: \"08d25537-fb6c-489f-83a9-aa3b0f73384c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9b45fb8fb-q97p8"
Sep 12 00:43:29.044437 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.044437 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1ff004e-ae36-49c3-a450-c718c377f171-installation-pull-secrets\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.044437 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-ca\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.044437 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044434    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.044837 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044501    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzqxj\" (UniqueName: \"kubernetes.io/projected/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-kube-api-access-jzqxj\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.044837 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3478ef0-9b7d-4705-b38d-7fdb72c084ad-tmp\") pod \"klusterlet-addon-workmgr-7445d698fb-tjvm6\" (UID: \"e3478ef0-9b7d-4705-b38d-7fdb72c084ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"
Sep 12 00:43:29.044837 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-config-volume\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:29.044837 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1ff004e-ae36-49c3-a450-c718c377f171-image-registry-private-configuration\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.044837 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.044837 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1ff004e-ae36-49c3-a450-c718c377f171-trusted-ca\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.044837 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1ff004e-ae36-49c3-a450-c718c377f171-ca-trust-extracted\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.044837 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fp25\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-kube-api-access-7fp25\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.044837 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1ff004e-ae36-49c3-a450-c718c377f171-registry-certificates\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.044837 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.044785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-bound-sa-token\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.140494 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.140427    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ntsr9_fae396c5-d77f-4f6e-9881-b69beadbc497/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:29.145443 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.145425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1ff004e-ae36-49c3-a450-c718c377f171-registry-certificates\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.145497 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.145449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-bound-sa-token\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.145497 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.145481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e3478ef0-9b7d-4705-b38d-7fdb72c084ad-klusterlet-config\") pod \"klusterlet-addon-workmgr-7445d698fb-tjvm6\" (UID: \"e3478ef0-9b7d-4705-b38d-7fdb72c084ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"
Sep 12 00:43:29.145657 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.145636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.145725 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.145665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:29.145725 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.145683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/08d25537-fb6c-489f-83a9-aa3b0f73384c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-9b45fb8fb-q97p8\" (UID: \"08d25537-fb6c-489f-83a9-aa3b0f73384c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9b45fb8fb-q97p8"
Sep 12 00:43:29.145725 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.145703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlfgf\" (UniqueName: \"kubernetes.io/projected/e3478ef0-9b7d-4705-b38d-7fdb72c084ad-kube-api-access-wlfgf\") pod \"klusterlet-addon-workmgr-7445d698fb-tjvm6\" (UID: \"e3478ef0-9b7d-4705-b38d-7fdb72c084ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"
Sep 12 00:43:29.145725 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.145718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-tmp-dir\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:29.145894 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.145742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-hub\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.145894 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.145779    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:43:29.145894 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.145838    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls podName:2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f nodeName:}" failed. No retries permitted until 2026-09-12 00:43:29.645822267 +0000 UTC m=+34.565114037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls") pod "dns-default-sbqdk" (UID: "2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f") : secret "dns-default-metrics-tls" not found
Sep 12 00:43:29.145894 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.145856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894hf\" (UniqueName: \"kubernetes.io/projected/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-kube-api-access-894hf\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:29.145894 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.145881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77hxf\" (UniqueName: \"kubernetes.io/projected/08d25537-fb6c-489f-83a9-aa3b0f73384c-kube-api-access-77hxf\") pod \"managed-serviceaccount-addon-agent-9b45fb8fb-q97p8\" (UID: \"08d25537-fb6c-489f-83a9-aa3b0f73384c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9b45fb8fb-q97p8"
Sep 12 00:43:29.146142 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.145909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.146142 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.145929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1ff004e-ae36-49c3-a450-c718c377f171-installation-pull-secrets\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.146142 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.145956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-ca\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.146142 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.145982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.146142 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.146020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zcw5\" (UniqueName: \"kubernetes.io/projected/b20511dd-b5f9-4302-b030-13bd00f31b12-kube-api-access-8zcw5\") pod \"ingress-canary-xbw45\" (UID: \"b20511dd-b5f9-4302-b030-13bd00f31b12\") " pod="openshift-ingress-canary/ingress-canary-xbw45"
Sep 12 00:43:29.146142 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.146061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqxj\" (UniqueName: \"kubernetes.io/projected/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-kube-api-access-jzqxj\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.146142 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.146107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3478ef0-9b7d-4705-b38d-7fdb72c084ad-tmp\") pod \"klusterlet-addon-workmgr-7445d698fb-tjvm6\" (UID: \"e3478ef0-9b7d-4705-b38d-7fdb72c084ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"
Sep 12 00:43:29.146142 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.146128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-config-volume\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:29.146514 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.146206    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-tmp-dir\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:29.146514 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.146505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1ff004e-ae36-49c3-a450-c718c377f171-image-registry-private-configuration\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.146642 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.146535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b20511dd-b5f9-4302-b030-13bd00f31b12-cert\") pod \"ingress-canary-xbw45\" (UID: \"b20511dd-b5f9-4302-b030-13bd00f31b12\") " pod="openshift-ingress-canary/ingress-canary-xbw45"
Sep 12 00:43:29.146642 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.146549    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.146642 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.146582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.146642 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.146619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1ff004e-ae36-49c3-a450-c718c377f171-trusted-ca\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.146810 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.146653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1ff004e-ae36-49c3-a450-c718c377f171-ca-trust-extracted\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.146810 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.146663    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 00:43:29.146810 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.146678    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7957768c6f-z2v8k: secret "image-registry-tls" not found
Sep 12 00:43:29.146810 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.146679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fp25\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-kube-api-access-7fp25\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.146810 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.146735    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls podName:a1ff004e-ae36-49c3-a450-c718c377f171 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:29.646719065 +0000 UTC m=+34.566010856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls") pod "image-registry-7957768c6f-z2v8k" (UID: "a1ff004e-ae36-49c3-a450-c718c377f171") : secret "image-registry-tls" not found
Sep 12 00:43:29.147404 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.147380    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1ff004e-ae36-49c3-a450-c718c377f171-registry-certificates\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.147404 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.147380    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1ff004e-ae36-49c3-a450-c718c377f171-ca-trust-extracted\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.148143 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.147684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1ff004e-ae36-49c3-a450-c718c377f171-trusted-ca\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.148143 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.147699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3478ef0-9b7d-4705-b38d-7fdb72c084ad-tmp\") pod \"klusterlet-addon-workmgr-7445d698fb-tjvm6\" (UID: \"e3478ef0-9b7d-4705-b38d-7fdb72c084ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"
Sep 12 00:43:29.149053 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.149029    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-config-volume\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:29.151080 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.151000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1ff004e-ae36-49c3-a450-c718c377f171-installation-pull-secrets\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.151080 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.151056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-hub\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.151080 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.151055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.151278 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.151134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e3478ef0-9b7d-4705-b38d-7fdb72c084ad-klusterlet-config\") pod \"klusterlet-addon-workmgr-7445d698fb-tjvm6\" (UID: \"e3478ef0-9b7d-4705-b38d-7fdb72c084ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"
Sep 12 00:43:29.151278 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.151150    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/08d25537-fb6c-489f-83a9-aa3b0f73384c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-9b45fb8fb-q97p8\" (UID: \"08d25537-fb6c-489f-83a9-aa3b0f73384c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9b45fb8fb-q97p8"
Sep 12 00:43:29.151278 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.151211    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.151278 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.151255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1ff004e-ae36-49c3-a450-c718c377f171-image-registry-private-configuration\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.152140 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.152119    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-ca\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.155096 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.155001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-bound-sa-token\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.155545 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.155525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlfgf\" (UniqueName: \"kubernetes.io/projected/e3478ef0-9b7d-4705-b38d-7fdb72c084ad-kube-api-access-wlfgf\") pod \"klusterlet-addon-workmgr-7445d698fb-tjvm6\" (UID: \"e3478ef0-9b7d-4705-b38d-7fdb72c084ad\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"
Sep 12 00:43:29.155758 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.155741    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzqxj\" (UniqueName: \"kubernetes.io/projected/68cf1131-fecb-4c45-ac7b-bf81e9de1cff-kube-api-access-jzqxj\") pod \"cluster-proxy-proxy-agent-7df6999d96-ndmqg\" (UID: \"68cf1131-fecb-4c45-ac7b-bf81e9de1cff\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.155844 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.155823    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fp25\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-kube-api-access-7fp25\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.156148 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.156130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-894hf\" (UniqueName: \"kubernetes.io/projected/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-kube-api-access-894hf\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:29.156210 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.156158    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77hxf\" (UniqueName: \"kubernetes.io/projected/08d25537-fb6c-489f-83a9-aa3b0f73384c-kube-api-access-77hxf\") pod \"managed-serviceaccount-addon-agent-9b45fb8fb-q97p8\" (UID: \"08d25537-fb6c-489f-83a9-aa3b0f73384c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9b45fb8fb-q97p8"
Sep 12 00:43:29.211898 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.211877    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9b45fb8fb-q97p8"
Sep 12 00:43:29.225043 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.225028    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:43:29.247036 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.246972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs\") pod \"network-metrics-daemon-k776z\" (UID: \"00498a5c-bd6a-4220-a017-53f01b8c1e92\") " pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:29.247036 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.247016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zcw5\" (UniqueName: \"kubernetes.io/projected/b20511dd-b5f9-4302-b030-13bd00f31b12-kube-api-access-8zcw5\") pod \"ingress-canary-xbw45\" (UID: \"b20511dd-b5f9-4302-b030-13bd00f31b12\") " pod="openshift-ingress-canary/ingress-canary-xbw45"
Sep 12 00:43:29.247141 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.247123    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:29.247197 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.247182    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs podName:00498a5c-bd6a-4220-a017-53f01b8c1e92 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:01.24716468 +0000 UTC m=+66.166456462 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs") pod "network-metrics-daemon-k776z" (UID: "00498a5c-bd6a-4220-a017-53f01b8c1e92") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 00:43:29.247258 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.247128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b20511dd-b5f9-4302-b030-13bd00f31b12-cert\") pod \"ingress-canary-xbw45\" (UID: \"b20511dd-b5f9-4302-b030-13bd00f31b12\") " pod="openshift-ingress-canary/ingress-canary-xbw45"
Sep 12 00:43:29.247308 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.247249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:29.247359 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.247303    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:29.247359 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.247348    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret podName:c34751eb-27f9-4061-8e37-d49bd4bb8bd5 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:01.247336885 +0000 UTC m=+66.166628650 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret") pod "global-pull-secret-syncer-nggqt" (UID: "c34751eb-27f9-4061-8e37-d49bd4bb8bd5") : object "kube-system"/"original-pull-secret" not registered
Sep 12 00:43:29.249149 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.249134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b20511dd-b5f9-4302-b030-13bd00f31b12-cert\") pod \"ingress-canary-xbw45\" (UID: \"b20511dd-b5f9-4302-b030-13bd00f31b12\") " pod="openshift-ingress-canary/ingress-canary-xbw45"
Sep 12 00:43:29.255173 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.255149    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zcw5\" (UniqueName: \"kubernetes.io/projected/b20511dd-b5f9-4302-b030-13bd00f31b12-kube-api-access-8zcw5\") pod \"ingress-canary-xbw45\" (UID: \"b20511dd-b5f9-4302-b030-13bd00f31b12\") " pod="openshift-ingress-canary/ingress-canary-xbw45"
Sep 12 00:43:29.285554 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.285493    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"
Sep 12 00:43:29.295589 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.295441    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xbw45"
Sep 12 00:43:29.348424 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.347818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqw7n\" (UniqueName: \"kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n\") pod \"network-check-target-v2z26\" (UID: \"b4e3082f-9a39-472f-b400-48312f862ba1\") " pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:29.348424 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.347999    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 00:43:29.348424 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.348013    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 00:43:29.348424 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.348025    2579 projected.go:196] Error preparing data for projected volume kube-api-access-cqw7n for pod openshift-network-diagnostics/network-check-target-v2z26: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:29.348424 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.348067    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n podName:b4e3082f-9a39-472f-b400-48312f862ba1 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:01.348055348 +0000 UTC m=+66.267347113 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqw7n" (UniqueName: "kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n") pod "network-check-target-v2z26" (UID: "b4e3082f-9a39-472f-b400-48312f862ba1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 00:43:29.526061 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.526035    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9b45fb8fb-q97p8"]
Sep 12 00:43:29.528864 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.528844    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"]
Sep 12 00:43:29.529559 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:43:29.529444    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08d25537_fb6c_489f_83a9_aa3b0f73384c.slice/crio-04444f6ae94b97e671010a8d5c6265198c2d3c61a7498fc929d268c6f76414c2 WatchSource:0}: Error finding container 04444f6ae94b97e671010a8d5c6265198c2d3c61a7498fc929d268c6f76414c2: Status 404 returned error can't find the container with id 04444f6ae94b97e671010a8d5c6265198c2d3c61a7498fc929d268c6f76414c2
Sep 12 00:43:29.531947 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:43:29.531925    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68cf1131_fecb_4c45_ac7b_bf81e9de1cff.slice/crio-79213e9713d0294c2c36a2d52365a1afdb543f41019066e826564f7ca0e98b66 WatchSource:0}: Error finding container 79213e9713d0294c2c36a2d52365a1afdb543f41019066e826564f7ca0e98b66: Status 404 returned error can't find the container with id 79213e9713d0294c2c36a2d52365a1afdb543f41019066e826564f7ca0e98b66
Sep 12 00:43:29.578680 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.578655    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"]
Sep 12 00:43:29.581473 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:43:29.581451    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3478ef0_9b7d_4705_b38d_7fdb72c084ad.slice/crio-6ef2c5669ac82fb89ada447bb8716402f279d5a2ed530bac2077526563d791f0 WatchSource:0}: Error finding container 6ef2c5669ac82fb89ada447bb8716402f279d5a2ed530bac2077526563d791f0: Status 404 returned error can't find the container with id 6ef2c5669ac82fb89ada447bb8716402f279d5a2ed530bac2077526563d791f0
Sep 12 00:43:29.585005 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.584983    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xbw45"]
Sep 12 00:43:29.591137 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:43:29.591117    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb20511dd_b5f9_4302_b030_13bd00f31b12.slice/crio-2a2b0088f0f92283232c59be66e4f179b1d464bb5973f13f2b5dd7743d9fd175 WatchSource:0}: Error finding container 2a2b0088f0f92283232c59be66e4f179b1d464bb5973f13f2b5dd7743d9fd175: Status 404 returned error can't find the container with id 2a2b0088f0f92283232c59be66e4f179b1d464bb5973f13f2b5dd7743d9fd175
Sep 12 00:43:29.591639 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.591619    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:43:29.593930 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.593915    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 12 00:43:29.594097 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.594070    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-42kqk\""
Sep 12 00:43:29.650183 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.650011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:29.650290 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.650198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:29.650290 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.650166    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:43:29.650372 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.650304    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 00:43:29.650372 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.650312    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls podName:2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f nodeName:}" failed. No retries permitted until 2026-09-12 00:43:30.650295939 +0000 UTC m=+35.569587723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls") pod "dns-default-sbqdk" (UID: "2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f") : secret "dns-default-metrics-tls" not found
Sep 12 00:43:29.650372 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.650313    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7957768c6f-z2v8k: secret "image-registry-tls" not found
Sep 12 00:43:29.650372 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:29.650341    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls podName:a1ff004e-ae36-49c3-a450-c718c377f171 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:30.650335265 +0000 UTC m=+35.569627035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls") pod "image-registry-7957768c6f-z2v8k" (UID: "a1ff004e-ae36-49c3-a450-c718c377f171") : secret "image-registry-tls" not found
Sep 12 00:43:29.747897 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.747871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg" event={"ID":"68cf1131-fecb-4c45-ac7b-bf81e9de1cff","Type":"ContainerStarted","Data":"79213e9713d0294c2c36a2d52365a1afdb543f41019066e826564f7ca0e98b66"}
Sep 12 00:43:29.749001 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.748981    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xbw45" event={"ID":"b20511dd-b5f9-4302-b030-13bd00f31b12","Type":"ContainerStarted","Data":"2a2b0088f0f92283232c59be66e4f179b1d464bb5973f13f2b5dd7743d9fd175"}
Sep 12 00:43:29.750037 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.750011    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9b45fb8fb-q97p8" event={"ID":"08d25537-fb6c-489f-83a9-aa3b0f73384c","Type":"ContainerStarted","Data":"04444f6ae94b97e671010a8d5c6265198c2d3c61a7498fc929d268c6f76414c2"}
Sep 12 00:43:29.752953 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.752930    2579 generic.go:356] "Generic (PLEG): container finished" podID="8fd9311d-6922-42f5-8689-e768402de953" containerID="4a3d419b434dc4a302cf679439685bf9d297e30aacb2d064bb7435f8440fe6b3" exitCode=0
Sep 12 00:43:29.753041 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.753008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4l4j" event={"ID":"8fd9311d-6922-42f5-8689-e768402de953","Type":"ContainerDied","Data":"4a3d419b434dc4a302cf679439685bf9d297e30aacb2d064bb7435f8440fe6b3"}
Sep 12 00:43:29.754098 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:29.754079    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6" event={"ID":"e3478ef0-9b7d-4705-b38d-7fdb72c084ad","Type":"ContainerStarted","Data":"6ef2c5669ac82fb89ada447bb8716402f279d5a2ed530bac2077526563d791f0"}
Sep 12 00:43:30.585109 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:30.585067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:43:30.585553 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:30.585488    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:43:30.590708 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:30.589632    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pfcnt\""
Sep 12 00:43:30.590708 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:30.589921    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 12 00:43:30.590708 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:30.590138    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 12 00:43:30.590708 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:30.590318    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 12 00:43:30.658224 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:30.658189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:30.658341 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:30.658244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:30.658398 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:30.658382    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 00:43:30.658431 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:30.658404    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7957768c6f-z2v8k: secret "image-registry-tls" not found
Sep 12 00:43:30.658500 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:30.658461    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls podName:a1ff004e-ae36-49c3-a450-c718c377f171 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:32.658442516 +0000 UTC m=+37.577734287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls") pod "image-registry-7957768c6f-z2v8k" (UID: "a1ff004e-ae36-49c3-a450-c718c377f171") : secret "image-registry-tls" not found
Sep 12 00:43:30.658895 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:30.658875    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:43:30.658987 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:30.658935    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls podName:2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f nodeName:}" failed. No retries permitted until 2026-09-12 00:43:32.658919437 +0000 UTC m=+37.578211207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls") pod "dns-default-sbqdk" (UID: "2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f") : secret "dns-default-metrics-tls" not found
Sep 12 00:43:30.762878 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:30.762846    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4l4j" event={"ID":"8fd9311d-6922-42f5-8689-e768402de953","Type":"ContainerStarted","Data":"8c1379e16df98d433c214280ff958f8398488a89d994610a7972b8e767bb5bc4"}
Sep 12 00:43:30.786695 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:30.786380    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g4l4j" podStartSLOduration=7.773988792 podStartE2EDuration="35.786358704s" totalImagesPullingTime="28.012369912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:42:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:30.785264916 +0000 UTC m=+35.704556730" watchObservedRunningTime="2026-09-12 00:43:30.786358704 +0000 UTC m=+35.705650494"
Sep 12 00:43:30.909184 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:30.909104    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ntsr9_fae396c5-d77f-4f6e-9881-b69beadbc497/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:30.976423 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:30.976391    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ntsr9_fae396c5-d77f-4f6e-9881-b69beadbc497/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:31.632510 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:31.632476    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ntsr9_fae396c5-d77f-4f6e-9881-b69beadbc497/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:31.695700 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:31.695590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ntsr9_fae396c5-d77f-4f6e-9881-b69beadbc497/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:32.673962 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:32.673927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:32.674522 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:32.674026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:32.674522 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:32.674106    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 00:43:32.674522 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:32.674136    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7957768c6f-z2v8k: secret "image-registry-tls" not found
Sep 12 00:43:32.674522 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:32.674203    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls podName:a1ff004e-ae36-49c3-a450-c718c377f171 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:36.674181829 +0000 UTC m=+41.593473610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls") pod "image-registry-7957768c6f-z2v8k" (UID: "a1ff004e-ae36-49c3-a450-c718c377f171") : secret "image-registry-tls" not found
Sep 12 00:43:32.674522 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:32.674116    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:43:32.674522 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:32.674256    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls podName:2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f nodeName:}" failed. No retries permitted until 2026-09-12 00:43:36.674243492 +0000 UTC m=+41.593535258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls") pod "dns-default-sbqdk" (UID: "2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f") : secret "dns-default-metrics-tls" not found
Sep 12 00:43:33.114082 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:33.113936    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ntsr9_fae396c5-d77f-4f6e-9881-b69beadbc497/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:33.214798 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:33.214741    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ntsr9_fae396c5-d77f-4f6e-9881-b69beadbc497/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:34.321787 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:34.321674    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ntsr9_fae396c5-d77f-4f6e-9881-b69beadbc497/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:35.775840 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:35.775805    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xbw45" event={"ID":"b20511dd-b5f9-4302-b030-13bd00f31b12","Type":"ContainerStarted","Data":"3a6f40302b6d15a74c351fc030a93b87cc99a8993060fcf65738136fd118fc20"}
Sep 12 00:43:35.777342 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:35.777318    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9b45fb8fb-q97p8" event={"ID":"08d25537-fb6c-489f-83a9-aa3b0f73384c","Type":"ContainerStarted","Data":"a79f6033eb12fd0950c28b3369e78f5d9eaeb7732ed7c13eb90d0aa19340e15b"}
Sep 12 00:43:35.779372 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:35.779318    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg" event={"ID":"68cf1131-fecb-4c45-ac7b-bf81e9de1cff","Type":"ContainerStarted","Data":"1d2c9cde45ad05a0a1ed73a54247a646d934e889f4fa7882d201d63cdf99b842"}
Sep 12 00:43:35.779372 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:35.779346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg" event={"ID":"68cf1131-fecb-4c45-ac7b-bf81e9de1cff","Type":"ContainerStarted","Data":"acb0a7de66aa787e29a192d4b5dcd5ca63440e30f6147553f8775964b1c1c1c6"}
Sep 12 00:43:35.779372 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:35.779359    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg" event={"ID":"68cf1131-fecb-4c45-ac7b-bf81e9de1cff","Type":"ContainerStarted","Data":"817a53be4381dc2b1b8331705218c6def750c4d2f2e6721b22f58055a35e65d0"}
Sep 12 00:43:35.792303 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:35.792259    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xbw45" podStartSLOduration=2.110434082 podStartE2EDuration="7.792246734s" totalImagesPullingTime="5.681812652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:35.791662785 +0000 UTC m=+40.710954575" watchObservedRunningTime="2026-09-12 00:43:35.792246734 +0000 UTC m=+40.711538538"
Sep 12 00:43:35.806433 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:35.806393    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-9b45fb8fb-q97p8" podStartSLOduration=12.063278789 podStartE2EDuration="17.806379511s" totalImagesPullingTime="5.743100722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:35.806098492 +0000 UTC m=+40.725390272" watchObservedRunningTime="2026-09-12 00:43:35.806379511 +0000 UTC m=+40.725671289"
Sep 12 00:43:35.825333 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:35.825274    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg" podStartSLOduration=12.068629545 podStartE2EDuration="17.825253373s" totalImagesPullingTime="5.756623828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:35.824869878 +0000 UTC m=+40.744161666" watchObservedRunningTime="2026-09-12 00:43:35.825253373 +0000 UTC m=+40.744545159"
Sep 12 00:43:36.013667 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:36.013631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ntsr9_fae396c5-d77f-4f6e-9881-b69beadbc497/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:36.112706 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:36.112622    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ntsr9_fae396c5-d77f-4f6e-9881-b69beadbc497/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:36.707494 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:36.707455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:36.707494 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:36.707498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:36.707720 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:36.707622    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:43:36.707720 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:36.707679    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 00:43:36.707720 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:36.707701    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7957768c6f-z2v8k: secret "image-registry-tls" not found
Sep 12 00:43:36.707813 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:36.707686    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls podName:2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f nodeName:}" failed. No retries permitted until 2026-09-12 00:43:44.707670687 +0000 UTC m=+49.626962451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls") pod "dns-default-sbqdk" (UID: "2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f") : secret "dns-default-metrics-tls" not found
Sep 12 00:43:36.707813 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:36.707752    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls podName:a1ff004e-ae36-49c3-a450-c718c377f171 nodeName:}" failed. No retries permitted until 2026-09-12 00:43:44.707739661 +0000 UTC m=+49.627031425 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls") pod "image-registry-7957768c6f-z2v8k" (UID: "a1ff004e-ae36-49c3-a450-c718c377f171") : secret "image-registry-tls" not found
Sep 12 00:43:37.082991 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:37.082882    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ntsr9_fae396c5-d77f-4f6e-9881-b69beadbc497/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:37.184037 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:37.183993    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ntsr9_fae396c5-d77f-4f6e-9881-b69beadbc497/konnectivity-agent/0.log" limit=1024
Sep 12 00:43:44.767917 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:44.767592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:43:44.767917 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:44.767692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:43:44.767917 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:44.767736    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:43:44.767917 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:44.767813    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls podName:2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f nodeName:}" failed. No retries permitted until 2026-09-12 00:44:00.767792099 +0000 UTC m=+65.687083871 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls") pod "dns-default-sbqdk" (UID: "2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f") : secret "dns-default-metrics-tls" not found
Sep 12 00:43:44.767917 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:44.767832    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 00:43:44.767917 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:44.767846    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7957768c6f-z2v8k: secret "image-registry-tls" not found
Sep 12 00:43:44.767917 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:43:44.767893    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls podName:a1ff004e-ae36-49c3-a450-c718c377f171 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:00.767877724 +0000 UTC m=+65.687169502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls") pod "image-registry-7957768c6f-z2v8k" (UID: "a1ff004e-ae36-49c3-a450-c718c377f171") : secret "image-registry-tls" not found
Sep 12 00:43:46.809045 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:46.808954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6" event={"ID":"e3478ef0-9b7d-4705-b38d-7fdb72c084ad","Type":"ContainerStarted","Data":"4f20e9fbc8d3ee6b8b23aab3efcc33bbe1588404fff0c25090984eedb0238470"}
Sep 12 00:43:46.809418 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:46.809101    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"
Sep 12 00:43:46.810557 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:46.810533    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6"
Sep 12 00:43:46.826086 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:46.826044    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7445d698fb-tjvm6" podStartSLOduration=11.937110911 podStartE2EDuration="28.826030384s" totalImagesPullingTime="16.888919473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:43:46.825552394 +0000 UTC m=+51.744844181" watchObservedRunningTime="2026-09-12 00:43:46.826030384 +0000 UTC m=+51.745322171"
Sep 12 00:43:54.743063 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:43:54.743030    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qzkgb"
Sep 12 00:44:00.780766 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:00.780732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:44:00.781124 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:00.780812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:44:00.781124 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:00.780884    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 00:44:00.781124 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:00.780908    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7957768c6f-z2v8k: secret "image-registry-tls" not found
Sep 12 00:44:00.781124 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:00.780957    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls podName:a1ff004e-ae36-49c3-a450-c718c377f171 nodeName:}" failed. No retries permitted until 2026-09-12 00:44:32.780941439 +0000 UTC m=+97.700233208 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls") pod "image-registry-7957768c6f-z2v8k" (UID: "a1ff004e-ae36-49c3-a450-c718c377f171") : secret "image-registry-tls" not found
Sep 12 00:44:00.781124 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:00.780892    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:44:00.781124 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:00.781052    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls podName:2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f nodeName:}" failed. No retries permitted until 2026-09-12 00:44:32.7810405 +0000 UTC m=+97.700332265 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls") pod "dns-default-sbqdk" (UID: "2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f") : secret "dns-default-metrics-tls" not found
Sep 12 00:44:01.284040 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.284010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:44:01.284208 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.284063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs\") pod \"network-metrics-daemon-k776z\" (UID: \"00498a5c-bd6a-4220-a017-53f01b8c1e92\") " pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:44:01.286729 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.286710    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 12 00:44:01.286828 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.286811    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 12 00:44:01.294627 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:01.294573    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 12 00:44:01.294727 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:01.294666    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs podName:00498a5c-bd6a-4220-a017-53f01b8c1e92 nodeName:}" failed. No retries permitted until 2026-09-12 00:45:05.294651228 +0000 UTC m=+130.213942998 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs") pod "network-metrics-daemon-k776z" (UID: "00498a5c-bd6a-4220-a017-53f01b8c1e92") : secret "metrics-daemon-secret" not found
Sep 12 00:44:01.297474 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.297458    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c34751eb-27f9-4061-8e37-d49bd4bb8bd5-original-pull-secret\") pod \"global-pull-secret-syncer-nggqt\" (UID: \"c34751eb-27f9-4061-8e37-d49bd4bb8bd5\") " pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:44:01.384662 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.384636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqw7n\" (UniqueName: \"kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n\") pod \"network-check-target-v2z26\" (UID: \"b4e3082f-9a39-472f-b400-48312f862ba1\") " pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:44:01.387740 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.387725    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 12 00:44:01.397243 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.397226    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 12 00:44:01.407643 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.407613    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqw7n\" (UniqueName: \"kubernetes.io/projected/b4e3082f-9a39-472f-b400-48312f862ba1-kube-api-access-cqw7n\") pod \"network-check-target-v2z26\" (UID: \"b4e3082f-9a39-472f-b400-48312f862ba1\") " pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:44:01.503527 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.503503    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nggqt"
Sep 12 00:44:01.513442 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.513419    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pfcnt\""
Sep 12 00:44:01.521369 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.521349    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:44:01.771853 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.771825    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nggqt"]
Sep 12 00:44:01.774961 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:44:01.774936    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34751eb_27f9_4061_8e37_d49bd4bb8bd5.slice/crio-a775f01ddc64bf2af0b84c29b09a2c694bee3e35aa08fb0dfab7dd4b4775a696 WatchSource:0}: Error finding container a775f01ddc64bf2af0b84c29b09a2c694bee3e35aa08fb0dfab7dd4b4775a696: Status 404 returned error can't find the container with id a775f01ddc64bf2af0b84c29b09a2c694bee3e35aa08fb0dfab7dd4b4775a696
Sep 12 00:44:01.793249 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.793220    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v2z26"]
Sep 12 00:44:01.796401 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:44:01.796379    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e3082f_9a39_472f_b400_48312f862ba1.slice/crio-ea86a4213c909bc6f843b6475c651645d2431466514b9f715a8eb678297d09f4 WatchSource:0}: Error finding container ea86a4213c909bc6f843b6475c651645d2431466514b9f715a8eb678297d09f4: Status 404 returned error can't find the container with id ea86a4213c909bc6f843b6475c651645d2431466514b9f715a8eb678297d09f4
Sep 12 00:44:01.853470 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.853398    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nggqt" event={"ID":"c34751eb-27f9-4061-8e37-d49bd4bb8bd5","Type":"ContainerStarted","Data":"a775f01ddc64bf2af0b84c29b09a2c694bee3e35aa08fb0dfab7dd4b4775a696"}
Sep 12 00:44:01.854561 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:01.854539    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v2z26" event={"ID":"b4e3082f-9a39-472f-b400-48312f862ba1","Type":"ContainerStarted","Data":"ea86a4213c909bc6f843b6475c651645d2431466514b9f715a8eb678297d09f4"}
Sep 12 00:44:06.877138 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:06.877052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v2z26" event={"ID":"b4e3082f-9a39-472f-b400-48312f862ba1","Type":"ContainerStarted","Data":"70fa4cd811892ef1b179d158d4fe2f0873e728bf2caf22dfa62fdb719ecf9fb5"}
Sep 12 00:44:06.877535 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:06.877196    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:44:06.878543 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:06.878522    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nggqt" event={"ID":"c34751eb-27f9-4061-8e37-d49bd4bb8bd5","Type":"ContainerStarted","Data":"a761a16d605be3b1ebd6bcc0a93953bf1acbb6a864982aa1811be7729296d898"}
Sep 12 00:44:06.894019 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:06.893979    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v2z26" podStartSLOduration=67.105586157 podStartE2EDuration="1m11.893968239s" totalImagesPullingTime="4.788382082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:06.892954199 +0000 UTC m=+71.812245986" watchObservedRunningTime="2026-09-12 00:44:06.893968239 +0000 UTC m=+71.813260026"
Sep 12 00:44:06.907914 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:06.907875    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nggqt" podStartSLOduration=66.092720158 podStartE2EDuration="1m10.907863803s" totalImagesPullingTime="4.815143645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:44:06.907547179 +0000 UTC m=+71.826838969" watchObservedRunningTime="2026-09-12 00:44:06.907863803 +0000 UTC m=+71.827155581"
Sep 12 00:44:32.802678 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:32.802646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:44:32.803024 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:32.802686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls\") pod \"image-registry-7957768c6f-z2v8k\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") " pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:44:32.803024 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:32.802799    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 00:44:32.803024 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:32.802869    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls podName:2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f nodeName:}" failed. No retries permitted until 2026-09-12 00:45:36.80285368 +0000 UTC m=+161.722145451 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls") pod "dns-default-sbqdk" (UID: "2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f") : secret "dns-default-metrics-tls" not found
Sep 12 00:44:32.803024 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:32.802802    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 12 00:44:32.803024 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:32.802905    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7957768c6f-z2v8k: secret "image-registry-tls" not found
Sep 12 00:44:32.803024 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:32.802953    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls podName:a1ff004e-ae36-49c3-a450-c718c377f171 nodeName:}" failed. No retries permitted until 2026-09-12 00:45:36.802941217 +0000 UTC m=+161.722232995 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls") pod "image-registry-7957768c6f-z2v8k" (UID: "a1ff004e-ae36-49c3-a450-c718c377f171") : secret "image-registry-tls" not found
Sep 12 00:44:37.886686 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:37.886650    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v2z26"
Sep 12 00:44:43.334260 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.334218    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4q7b5"]
Sep 12 00:44:43.337583 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.337556    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:43.340301 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.340273    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 12 00:44:43.340301 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.340294    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 12 00:44:43.340561 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.340545    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 12 00:44:43.340636 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.340621    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d5lqq\""
Sep 12 00:44:43.341167 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.341152    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 12 00:44:43.347654 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.347635    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4q7b5"]
Sep 12 00:44:43.474572 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.474545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-data-volume\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:43.474728 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.474585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:43.474728 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.474623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-crio-socket\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:43.474728 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.474683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:43.474834 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.474758    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt5td\" (UniqueName: \"kubernetes.io/projected/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-api-access-tt5td\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:43.575834 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.575811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:43.575925 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.575839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5td\" (UniqueName: \"kubernetes.io/projected/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-api-access-tt5td\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:43.575925 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.575915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-data-volume\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:43.575993 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:43.575927    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:44:43.575993 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.575947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:43.575993 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.575981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-crio-socket\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:43.576099 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:43.576005    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm podName:e37e550b-9235-4dd2-83e4-5e6fee7d2a9a nodeName:}" failed. No retries permitted until 2026-09-12 00:44:44.075986596 +0000 UTC m=+108.995278380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4q7b5" (UID: "e37e550b-9235-4dd2-83e4-5e6fee7d2a9a") : configmap "kube-rbac-proxy" not found
Sep 12 00:44:43.576099 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.576039    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-crio-socket\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:43.576190 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:43.576116    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:43.576239 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:43.576198    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls podName:e37e550b-9235-4dd2-83e4-5e6fee7d2a9a nodeName:}" failed. No retries permitted until 2026-09-12 00:44:44.076179361 +0000 UTC m=+108.995471126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4q7b5" (UID: "e37e550b-9235-4dd2-83e4-5e6fee7d2a9a") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:43.576288 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.576252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-data-volume\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:43.586925 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:43.586861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt5td\" (UniqueName: \"kubernetes.io/projected/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-api-access-tt5td\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:44.079820 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:44.079777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:44.080022 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:44.079835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:44.080022 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:44.079925    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:44:44.080022 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:44.079935    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:44.080022 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:44.079993    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm podName:e37e550b-9235-4dd2-83e4-5e6fee7d2a9a nodeName:}" failed. No retries permitted until 2026-09-12 00:44:45.079974262 +0000 UTC m=+109.999266040 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4q7b5" (UID: "e37e550b-9235-4dd2-83e4-5e6fee7d2a9a") : configmap "kube-rbac-proxy" not found
Sep 12 00:44:44.080022 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:44.080009    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls podName:e37e550b-9235-4dd2-83e4-5e6fee7d2a9a nodeName:}" failed. No retries permitted until 2026-09-12 00:44:45.080001328 +0000 UTC m=+109.999293095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4q7b5" (UID: "e37e550b-9235-4dd2-83e4-5e6fee7d2a9a") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:45.087718 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:45.087687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:45.088074 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:45.087737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:45.088074 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:45.087833    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:45.088074 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:45.087865    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:44:45.088074 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:45.087888    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls podName:e37e550b-9235-4dd2-83e4-5e6fee7d2a9a nodeName:}" failed. No retries permitted until 2026-09-12 00:44:47.087875519 +0000 UTC m=+112.007167287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4q7b5" (UID: "e37e550b-9235-4dd2-83e4-5e6fee7d2a9a") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:45.088074 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:45.087913    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm podName:e37e550b-9235-4dd2-83e4-5e6fee7d2a9a nodeName:}" failed. No retries permitted until 2026-09-12 00:44:47.087900922 +0000 UTC m=+112.007192703 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4q7b5" (UID: "e37e550b-9235-4dd2-83e4-5e6fee7d2a9a") : configmap "kube-rbac-proxy" not found
Sep 12 00:44:45.897669 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:45.897641    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6q7h9_64c9aebc-9e60-472a-891a-7853806abd9e/dns-node-resolver/0.log"
Sep 12 00:44:46.897744 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:46.897720    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7m4gn_305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5/node-ca/0.log"
Sep 12 00:44:47.100913 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:47.100881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:47.101081 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:47.100919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:47.101081 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:47.100984    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:44:47.101081 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:47.101016    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:47.101081 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:47.101032    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm podName:e37e550b-9235-4dd2-83e4-5e6fee7d2a9a nodeName:}" failed. No retries permitted until 2026-09-12 00:44:51.101018687 +0000 UTC m=+116.020310466 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4q7b5" (UID: "e37e550b-9235-4dd2-83e4-5e6fee7d2a9a") : configmap "kube-rbac-proxy" not found
Sep 12 00:44:47.101081 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:47.101064    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls podName:e37e550b-9235-4dd2-83e4-5e6fee7d2a9a nodeName:}" failed. No retries permitted until 2026-09-12 00:44:51.101051766 +0000 UTC m=+116.020343531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4q7b5" (UID: "e37e550b-9235-4dd2-83e4-5e6fee7d2a9a") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:48.098058 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:48.098031    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xbw45_b20511dd-b5f9-4302-b030-13bd00f31b12/serve-healthcheck-canary/0.log"
Sep 12 00:44:51.129557 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:51.129525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:51.129941 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:51.129564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:51.129941 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:51.129660    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 00:44:51.129941 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:51.129686    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:51.129941 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:51.129721    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm podName:e37e550b-9235-4dd2-83e4-5e6fee7d2a9a nodeName:}" failed. No retries permitted until 2026-09-12 00:44:59.129705752 +0000 UTC m=+124.048997518 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4q7b5" (UID: "e37e550b-9235-4dd2-83e4-5e6fee7d2a9a") : configmap "kube-rbac-proxy" not found
Sep 12 00:44:51.129941 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:44:51.129743    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls podName:e37e550b-9235-4dd2-83e4-5e6fee7d2a9a nodeName:}" failed. No retries permitted until 2026-09-12 00:44:59.129730879 +0000 UTC m=+124.049022649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4q7b5" (UID: "e37e550b-9235-4dd2-83e4-5e6fee7d2a9a") : secret "insights-runtime-extractor-tls" not found
Sep 12 00:44:59.196133 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:59.196094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:59.196589 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:59.196146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:59.196815 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:59.196793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:59.198318 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:59.198299    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e37e550b-9235-4dd2-83e4-5e6fee7d2a9a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4q7b5\" (UID: \"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a\") " pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:59.248947 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:59.248921    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d5lqq\""
Sep 12 00:44:59.257040 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:59.257022    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4q7b5"
Sep 12 00:44:59.525068 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:44:59.524995    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4q7b5"]
Sep 12 00:44:59.528981 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:44:59.528951    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37e550b_9235_4dd2_83e4_5e6fee7d2a9a.slice/crio-1b790988c25caa7ab816845fc7224ec86d04496acf2d17185fe7b08b016b18ce WatchSource:0}: Error finding container 1b790988c25caa7ab816845fc7224ec86d04496acf2d17185fe7b08b016b18ce: Status 404 returned error can't find the container with id 1b790988c25caa7ab816845fc7224ec86d04496acf2d17185fe7b08b016b18ce
Sep 12 00:45:00.032695 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:00.032663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q7b5" event={"ID":"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a","Type":"ContainerStarted","Data":"fcd8e29c466c44b65a36bbe1f76656942e0798ce985b4e1e3ab715d7ea5f0f1c"}
Sep 12 00:45:00.032862 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:00.032719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q7b5" event={"ID":"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a","Type":"ContainerStarted","Data":"1b790988c25caa7ab816845fc7224ec86d04496acf2d17185fe7b08b016b18ce"}
Sep 12 00:45:01.037846 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:01.037802    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q7b5" event={"ID":"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a","Type":"ContainerStarted","Data":"43a4e072d4244441bf1faf24bc59be2b75ef366480cc745844192339b186c37e"}
Sep 12 00:45:02.042726 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:02.042649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4q7b5" event={"ID":"e37e550b-9235-4dd2-83e4-5e6fee7d2a9a","Type":"ContainerStarted","Data":"cf4d05a0889d765e0c1680ff154bf2de679c64f6b87cdbbda5415c7cf0fc3caf"}
Sep 12 00:45:02.061205 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:02.061163    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4q7b5" podStartSLOduration=17.035513517 podStartE2EDuration="19.061147796s" totalImagesPullingTime="2.025634279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 00:44:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:02.060177219 +0000 UTC m=+126.979469007" watchObservedRunningTime="2026-09-12 00:45:02.061147796 +0000 UTC m=+126.980439583"
Sep 12 00:45:05.341162 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:05.341129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs\") pod \"network-metrics-daemon-k776z\" (UID: \"00498a5c-bd6a-4220-a017-53f01b8c1e92\") " pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:45:05.343281 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:05.343258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00498a5c-bd6a-4220-a017-53f01b8c1e92-metrics-certs\") pod \"network-metrics-daemon-k776z\" (UID: \"00498a5c-bd6a-4220-a017-53f01b8c1e92\") " pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:45:05.603912 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:05.603842    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-42kqk\""
Sep 12 00:45:05.612180 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:05.612155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k776z"
Sep 12 00:45:05.880175 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:05.880108    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k776z"]
Sep 12 00:45:05.882994 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:45:05.882968    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00498a5c_bd6a_4220_a017_53f01b8c1e92.slice/crio-8d78f67d71c036431bf79e9b6337045aee5e5af6178654f6408a68947295a883 WatchSource:0}: Error finding container 8d78f67d71c036431bf79e9b6337045aee5e5af6178654f6408a68947295a883: Status 404 returned error can't find the container with id 8d78f67d71c036431bf79e9b6337045aee5e5af6178654f6408a68947295a883
Sep 12 00:45:06.057792 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:06.057759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k776z" event={"ID":"00498a5c-bd6a-4220-a017-53f01b8c1e92","Type":"ContainerStarted","Data":"8d78f67d71c036431bf79e9b6337045aee5e5af6178654f6408a68947295a883"}
Sep 12 00:45:08.066067 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:08.066032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k776z" event={"ID":"00498a5c-bd6a-4220-a017-53f01b8c1e92","Type":"ContainerStarted","Data":"2ebdc0a0a0ef37faef5fecb06f9e98defa12cb8e78e17cfeea240db7b34720fa"}
Sep 12 00:45:08.066432 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:08.066084    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k776z" event={"ID":"00498a5c-bd6a-4220-a017-53f01b8c1e92","Type":"ContainerStarted","Data":"f8599fa5292cb2fd717791691ad9c9b8fcdf8d55ad08c3a41530fc09bfd5ba34"}
Sep 12 00:45:08.090976 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:08.089787    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-k776z" podStartSLOduration=131.943004217 podStartE2EDuration="2m13.089768171s" totalImagesPullingTime="1.146763954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:08.087838862 +0000 UTC m=+133.007130643" watchObservedRunningTime="2026-09-12 00:45:08.089768171 +0000 UTC m=+133.009059959"
Sep 12 00:45:09.226731 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:09.226687    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg" podUID="68cf1131-fecb-4c45-ac7b-bf81e9de1cff" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 12 00:45:19.226487 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.226443    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg" podUID="68cf1131-fecb-4c45-ac7b-bf81e9de1cff" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 12 00:45:19.842312 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.842280    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8xb2p"]
Sep 12 00:45:19.845732 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.845715    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:19.848226 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.848202    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 12 00:45:19.848405 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.848389    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-45bhm\""
Sep 12 00:45:19.849163 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.849142    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 12 00:45:19.849240 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.849142    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 12 00:45:19.849240 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.849197    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 12 00:45:19.849353 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.849175    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 12 00:45:19.849353 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.849168    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 12 00:45:19.940778 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.940755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5071b60a-fd23-44b4-94a4-9fbebd984ebc-sys\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:19.940884 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.940784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h2x9\" (UniqueName: \"kubernetes.io/projected/5071b60a-fd23-44b4-94a4-9fbebd984ebc-kube-api-access-9h2x9\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:19.940884 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.940806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5071b60a-fd23-44b4-94a4-9fbebd984ebc-node-exporter-tls\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:19.940884 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.940829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5071b60a-fd23-44b4-94a4-9fbebd984ebc-node-exporter-wtmp\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:19.941064 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.940904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5071b60a-fd23-44b4-94a4-9fbebd984ebc-root\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:19.941064 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.940942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5071b60a-fd23-44b4-94a4-9fbebd984ebc-node-exporter-textfile\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:19.941064 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.940979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5071b60a-fd23-44b4-94a4-9fbebd984ebc-node-exporter-accelerators-collector-config\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:19.941064 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.941007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5071b60a-fd23-44b4-94a4-9fbebd984ebc-metrics-client-ca\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:19.941064 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:19.941059    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5071b60a-fd23-44b4-94a4-9fbebd984ebc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.041931 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.041904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5071b60a-fd23-44b4-94a4-9fbebd984ebc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.042043 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.041948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5071b60a-fd23-44b4-94a4-9fbebd984ebc-sys\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.042043 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.041975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h2x9\" (UniqueName: \"kubernetes.io/projected/5071b60a-fd23-44b4-94a4-9fbebd984ebc-kube-api-access-9h2x9\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.042043 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.042007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5071b60a-fd23-44b4-94a4-9fbebd984ebc-node-exporter-tls\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.042043 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.042013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5071b60a-fd23-44b4-94a4-9fbebd984ebc-sys\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.042043 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.042042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5071b60a-fd23-44b4-94a4-9fbebd984ebc-node-exporter-wtmp\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.042291 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.042153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5071b60a-fd23-44b4-94a4-9fbebd984ebc-root\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.042291 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.042182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5071b60a-fd23-44b4-94a4-9fbebd984ebc-node-exporter-textfile\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.042291 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.042206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5071b60a-fd23-44b4-94a4-9fbebd984ebc-node-exporter-accelerators-collector-config\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.042291 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.042208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5071b60a-fd23-44b4-94a4-9fbebd984ebc-node-exporter-wtmp\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.042291 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.042232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5071b60a-fd23-44b4-94a4-9fbebd984ebc-metrics-client-ca\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.042291 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.042251    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5071b60a-fd23-44b4-94a4-9fbebd984ebc-root\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.042579 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.042527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5071b60a-fd23-44b4-94a4-9fbebd984ebc-node-exporter-textfile\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.042813 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.042792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5071b60a-fd23-44b4-94a4-9fbebd984ebc-node-exporter-accelerators-collector-config\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.042872 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.042808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5071b60a-fd23-44b4-94a4-9fbebd984ebc-metrics-client-ca\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.044301 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.044281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5071b60a-fd23-44b4-94a4-9fbebd984ebc-node-exporter-tls\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.044371 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.044298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5071b60a-fd23-44b4-94a4-9fbebd984ebc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.051095 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.051078    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h2x9\" (UniqueName: \"kubernetes.io/projected/5071b60a-fd23-44b4-94a4-9fbebd984ebc-kube-api-access-9h2x9\") pod \"node-exporter-8xb2p\" (UID: \"5071b60a-fd23-44b4-94a4-9fbebd984ebc\") " pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.154787 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:20.154708    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8xb2p"
Sep 12 00:45:20.166828 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:45:20.166796    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5071b60a_fd23_44b4_94a4_9fbebd984ebc.slice/crio-8dacebfc427962b6f40f1bb33a2182b1cc8a31b28ae77641dcb13fe128fed649 WatchSource:0}: Error finding container 8dacebfc427962b6f40f1bb33a2182b1cc8a31b28ae77641dcb13fe128fed649: Status 404 returned error can't find the container with id 8dacebfc427962b6f40f1bb33a2182b1cc8a31b28ae77641dcb13fe128fed649
Sep 12 00:45:21.106295 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:21.106265    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8xb2p" event={"ID":"5071b60a-fd23-44b4-94a4-9fbebd984ebc","Type":"ContainerStarted","Data":"8dacebfc427962b6f40f1bb33a2182b1cc8a31b28ae77641dcb13fe128fed649"}
Sep 12 00:45:22.110886 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:22.110847    2579 generic.go:356] "Generic (PLEG): container finished" podID="5071b60a-fd23-44b4-94a4-9fbebd984ebc" containerID="9844e448dbbf5d2aca8df224eab9b2fb70998768e58915f02207d657977f05a6" exitCode=0
Sep 12 00:45:22.111349 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:22.110907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8xb2p" event={"ID":"5071b60a-fd23-44b4-94a4-9fbebd984ebc","Type":"ContainerDied","Data":"9844e448dbbf5d2aca8df224eab9b2fb70998768e58915f02207d657977f05a6"}
Sep 12 00:45:23.116591 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:23.116555    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8xb2p" event={"ID":"5071b60a-fd23-44b4-94a4-9fbebd984ebc","Type":"ContainerStarted","Data":"599fcb6acc7b8f95cebab155561901fba4dbdf3049f1e0dad57e7f4d13d374f5"}
Sep 12 00:45:23.116959 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:23.116624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8xb2p" event={"ID":"5071b60a-fd23-44b4-94a4-9fbebd984ebc","Type":"ContainerStarted","Data":"5f12ccd8dc87178f79a79a9620c622fcc35c6825cc22734e747122cb265a2b86"}
Sep 12 00:45:23.163424 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:23.163377    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8xb2p" podStartSLOduration=3.198983907 podStartE2EDuration="4.163362609s" totalImagesPullingTime="905.287322ms" totalInitContainerRuntime="59.09138ms" isStatefulPod=true podCreationTimestamp="2026-09-12 00:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:23.161121375 +0000 UTC m=+148.080413162" watchObservedRunningTime="2026-09-12 00:45:23.163362609 +0000 UTC m=+148.082654396"
Sep 12 00:45:29.226100 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:29.226059    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg" podUID="68cf1131-fecb-4c45-ac7b-bf81e9de1cff" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 12 00:45:29.226542 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:29.226137    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg"
Sep 12 00:45:29.226619 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:29.226549    2579 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"1d2c9cde45ad05a0a1ed73a54247a646d934e889f4fa7882d201d63cdf99b842"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 12 00:45:29.226619 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:29.226595    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg" podUID="68cf1131-fecb-4c45-ac7b-bf81e9de1cff" containerName="service-proxy" containerID="cri-o://1d2c9cde45ad05a0a1ed73a54247a646d934e889f4fa7882d201d63cdf99b842" gracePeriod=30
Sep 12 00:45:30.141308 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:30.141271    2579 generic.go:356] "Generic (PLEG): container finished" podID="68cf1131-fecb-4c45-ac7b-bf81e9de1cff" containerID="1d2c9cde45ad05a0a1ed73a54247a646d934e889f4fa7882d201d63cdf99b842" exitCode=2
Sep 12 00:45:30.141478 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:30.141342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg" event={"ID":"68cf1131-fecb-4c45-ac7b-bf81e9de1cff","Type":"ContainerDied","Data":"1d2c9cde45ad05a0a1ed73a54247a646d934e889f4fa7882d201d63cdf99b842"}
Sep 12 00:45:30.141478 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:30.141393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7df6999d96-ndmqg" event={"ID":"68cf1131-fecb-4c45-ac7b-bf81e9de1cff","Type":"ContainerStarted","Data":"e5d9ba4146cf904dd2e16a635c2f1dae6f99d03c731aedc33e0ce11bc924f6ca"}
Sep 12 00:45:30.636365 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:30.636335    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7957768c6f-z2v8k"]
Sep 12 00:45:30.636728 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:45:30.636503    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7957768c6f-z2v8k" podUID="a1ff004e-ae36-49c3-a450-c718c377f171"
Sep 12 00:45:31.144404 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.144368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:45:31.151102 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.151081    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:45:31.225666 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.225646    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1ff004e-ae36-49c3-a450-c718c377f171-registry-certificates\") pod \"a1ff004e-ae36-49c3-a450-c718c377f171\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") "
Sep 12 00:45:31.225793 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.225681    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1ff004e-ae36-49c3-a450-c718c377f171-trusted-ca\") pod \"a1ff004e-ae36-49c3-a450-c718c377f171\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") "
Sep 12 00:45:31.225793 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.225726    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1ff004e-ae36-49c3-a450-c718c377f171-ca-trust-extracted\") pod \"a1ff004e-ae36-49c3-a450-c718c377f171\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") "
Sep 12 00:45:31.225793 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.225768    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1ff004e-ae36-49c3-a450-c718c377f171-image-registry-private-configuration\") pod \"a1ff004e-ae36-49c3-a450-c718c377f171\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") "
Sep 12 00:45:31.225950 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.225793    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-bound-sa-token\") pod \"a1ff004e-ae36-49c3-a450-c718c377f171\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") "
Sep 12 00:45:31.225950 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.225819    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1ff004e-ae36-49c3-a450-c718c377f171-installation-pull-secrets\") pod \"a1ff004e-ae36-49c3-a450-c718c377f171\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") "
Sep 12 00:45:31.225950 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.225863    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fp25\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-kube-api-access-7fp25\") pod \"a1ff004e-ae36-49c3-a450-c718c377f171\" (UID: \"a1ff004e-ae36-49c3-a450-c718c377f171\") "
Sep 12 00:45:31.226089 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.226069    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ff004e-ae36-49c3-a450-c718c377f171-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a1ff004e-ae36-49c3-a450-c718c377f171" (UID: "a1ff004e-ae36-49c3-a450-c718c377f171"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:45:31.226126 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.226085    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ff004e-ae36-49c3-a450-c718c377f171-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a1ff004e-ae36-49c3-a450-c718c377f171" (UID: "a1ff004e-ae36-49c3-a450-c718c377f171"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:45:31.226171 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.226124    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ff004e-ae36-49c3-a450-c718c377f171-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a1ff004e-ae36-49c3-a450-c718c377f171" (UID: "a1ff004e-ae36-49c3-a450-c718c377f171"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:45:31.228092 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.228058    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ff004e-ae36-49c3-a450-c718c377f171-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a1ff004e-ae36-49c3-a450-c718c377f171" (UID: "a1ff004e-ae36-49c3-a450-c718c377f171"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:45:31.228198 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.228080    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a1ff004e-ae36-49c3-a450-c718c377f171" (UID: "a1ff004e-ae36-49c3-a450-c718c377f171"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:45:31.228198 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.228101    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ff004e-ae36-49c3-a450-c718c377f171-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a1ff004e-ae36-49c3-a450-c718c377f171" (UID: "a1ff004e-ae36-49c3-a450-c718c377f171"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:45:31.228198 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.228141    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-kube-api-access-7fp25" (OuterVolumeSpecName: "kube-api-access-7fp25") pod "a1ff004e-ae36-49c3-a450-c718c377f171" (UID: "a1ff004e-ae36-49c3-a450-c718c377f171"). InnerVolumeSpecName "kube-api-access-7fp25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:45:31.326520 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.326496    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fp25\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-kube-api-access-7fp25\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:45:31.326520 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.326516    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1ff004e-ae36-49c3-a450-c718c377f171-registry-certificates\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:45:31.326520 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.326525    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1ff004e-ae36-49c3-a450-c718c377f171-trusted-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:45:31.326706 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.326534    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1ff004e-ae36-49c3-a450-c718c377f171-ca-trust-extracted\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:45:31.326706 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.326544    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1ff004e-ae36-49c3-a450-c718c377f171-image-registry-private-configuration\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:45:31.326706 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.326552    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-bound-sa-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:45:31.326706 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:31.326560    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1ff004e-ae36-49c3-a450-c718c377f171-installation-pull-secrets\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:45:31.990568 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:45:31.990530    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-sbqdk" podUID="2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f"
Sep 12 00:45:32.147652 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:32.147618    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:45:32.147822 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:32.147621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7957768c6f-z2v8k"
Sep 12 00:45:32.182717 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:32.182683    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7957768c6f-z2v8k"]
Sep 12 00:45:32.188534 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:32.188513    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7957768c6f-z2v8k"]
Sep 12 00:45:32.231557 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:32.231535    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1ff004e-ae36-49c3-a450-c718c377f171-registry-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:45:33.589147 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:33.589111    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1ff004e-ae36-49c3-a450-c718c377f171" path="/var/lib/kubelet/pods/a1ff004e-ae36-49c3-a450-c718c377f171/volumes"
Sep 12 00:45:36.866928 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:36.866892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:45:36.869068 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:36.869045    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f-metrics-tls\") pod \"dns-default-sbqdk\" (UID: \"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f\") " pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:45:36.950465 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:36.950439    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2gxdg\""
Sep 12 00:45:36.958433 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:36.958416    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:45:37.228245 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:37.228213    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sbqdk"]
Sep 12 00:45:37.231005 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:45:37.230977    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e96ed3a_ab87_4d37_a3a6_ccce53b09a3f.slice/crio-a4b0b95c9f41acdda233ef286068ba714f13923ef3792a07e598b72fbf4152c4 WatchSource:0}: Error finding container a4b0b95c9f41acdda233ef286068ba714f13923ef3792a07e598b72fbf4152c4: Status 404 returned error can't find the container with id a4b0b95c9f41acdda233ef286068ba714f13923ef3792a07e598b72fbf4152c4
Sep 12 00:45:38.166529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:38.166485    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sbqdk" event={"ID":"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f","Type":"ContainerStarted","Data":"a4b0b95c9f41acdda233ef286068ba714f13923ef3792a07e598b72fbf4152c4"}
Sep 12 00:45:40.175007 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:40.174968    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sbqdk" event={"ID":"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f","Type":"ContainerStarted","Data":"e47b5d2c63ceb0d96edde58554081223c1ea53de79cba771089955155194a81d"}
Sep 12 00:45:40.175373 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:40.175033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sbqdk" event={"ID":"2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f","Type":"ContainerStarted","Data":"bdac318e4730ab291ee1647d93c71ca3e2f06a2db8bbf8186777bdb072169e5c"}
Sep 12 00:45:40.175373 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:40.175062    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:45:40.193062 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:40.193014    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sbqdk" podStartSLOduration=130.32542633 podStartE2EDuration="2m12.19299747s" totalImagesPullingTime="1.86757114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:45:40.191135922 +0000 UTC m=+165.110427708" watchObservedRunningTime="2026-09-12 00:45:40.19299747 +0000 UTC m=+165.112289257"
Sep 12 00:45:50.181029 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:45:50.180955    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sbqdk"
Sep 12 00:47:55.499806 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:47:55.499774    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 00:47:55.501833 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:47:55.501810    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 00:47:55.503860 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:47:55.503840    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 12 00:48:19.245187 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:19.245154    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5rx8b"]
Sep 12 00:48:19.246910 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:19.246895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5rx8b"
Sep 12 00:48:19.250152 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:19.250136    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qfjbv\""
Sep 12 00:48:19.250295 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:19.250266    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 12 00:48:19.251161 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:19.251143    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 12 00:48:19.257078 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:19.257058    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5rx8b"]
Sep 12 00:48:19.314701 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:19.314673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a5936b3-cc56-4c83-8864-0329993e3056-bound-sa-token\") pod \"cert-manager-55b4d954d5-5rx8b\" (UID: \"1a5936b3-cc56-4c83-8864-0329993e3056\") " pod="cert-manager/cert-manager-55b4d954d5-5rx8b"
Sep 12 00:48:19.314855 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:19.314726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwcqc\" (UniqueName: \"kubernetes.io/projected/1a5936b3-cc56-4c83-8864-0329993e3056-kube-api-access-fwcqc\") pod \"cert-manager-55b4d954d5-5rx8b\" (UID: \"1a5936b3-cc56-4c83-8864-0329993e3056\") " pod="cert-manager/cert-manager-55b4d954d5-5rx8b"
Sep 12 00:48:19.415037 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:19.415005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwcqc\" (UniqueName: \"kubernetes.io/projected/1a5936b3-cc56-4c83-8864-0329993e3056-kube-api-access-fwcqc\") pod \"cert-manager-55b4d954d5-5rx8b\" (UID: \"1a5936b3-cc56-4c83-8864-0329993e3056\") " pod="cert-manager/cert-manager-55b4d954d5-5rx8b"
Sep 12 00:48:19.415181 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:19.415062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a5936b3-cc56-4c83-8864-0329993e3056-bound-sa-token\") pod \"cert-manager-55b4d954d5-5rx8b\" (UID: \"1a5936b3-cc56-4c83-8864-0329993e3056\") " pod="cert-manager/cert-manager-55b4d954d5-5rx8b"
Sep 12 00:48:19.423464 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:19.423436    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a5936b3-cc56-4c83-8864-0329993e3056-bound-sa-token\") pod \"cert-manager-55b4d954d5-5rx8b\" (UID: \"1a5936b3-cc56-4c83-8864-0329993e3056\") " pod="cert-manager/cert-manager-55b4d954d5-5rx8b"
Sep 12 00:48:19.423663 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:19.423643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwcqc\" (UniqueName: \"kubernetes.io/projected/1a5936b3-cc56-4c83-8864-0329993e3056-kube-api-access-fwcqc\") pod \"cert-manager-55b4d954d5-5rx8b\" (UID: \"1a5936b3-cc56-4c83-8864-0329993e3056\") " pod="cert-manager/cert-manager-55b4d954d5-5rx8b"
Sep 12 00:48:19.556058 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:19.555975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5rx8b"
Sep 12 00:48:19.823946 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:19.823873    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5rx8b"]
Sep 12 00:48:19.827169 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:48:19.827141    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a5936b3_cc56_4c83_8864_0329993e3056.slice/crio-967ef560f042ab26cd54cf944bf1eb7cc61f6a0f5b0c2fc88d36d1770042d95f WatchSource:0}: Error finding container 967ef560f042ab26cd54cf944bf1eb7cc61f6a0f5b0c2fc88d36d1770042d95f: Status 404 returned error can't find the container with id 967ef560f042ab26cd54cf944bf1eb7cc61f6a0f5b0c2fc88d36d1770042d95f
Sep 12 00:48:19.829082 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:19.829066    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:48:20.683579 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:20.683536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5rx8b" event={"ID":"1a5936b3-cc56-4c83-8864-0329993e3056","Type":"ContainerStarted","Data":"967ef560f042ab26cd54cf944bf1eb7cc61f6a0f5b0c2fc88d36d1770042d95f"}
Sep 12 00:48:23.694306 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:23.694272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5rx8b" event={"ID":"1a5936b3-cc56-4c83-8864-0329993e3056","Type":"ContainerStarted","Data":"d76fd61b2234e7178cdc4d69807c898dc3696cab53146417546c41cb7c0d90ec"}
Sep 12 00:48:23.713495 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:23.713451    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-5rx8b" podStartSLOduration=1.8720054849999999 podStartE2EDuration="4.713437428s" totalImagesPullingTime="2.841431943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:48:23.713189324 +0000 UTC m=+328.632481112" watchObservedRunningTime="2026-09-12 00:48:23.713437428 +0000 UTC m=+328.632729215"
Sep 12 00:48:42.493818 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.493785    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"]
Sep 12 00:48:42.497835 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.497816    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.501217 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.501198    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 12 00:48:42.501373 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.501238    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 12 00:48:42.501472 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.501456    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lr2d4\""
Sep 12 00:48:42.501537 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.501457    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 12 00:48:42.502181 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.502167    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 12 00:48:42.515956 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.515933    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"]
Sep 12 00:48:42.569583 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.569558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjbhp\" (UniqueName: \"kubernetes.io/projected/295af638-35e1-40a6-ab09-8f27213a7cbc-kube-api-access-fjbhp\") pod \"opendatahub-operator-controller-manager-694bbb96f5-z5mv2\" (UID: \"295af638-35e1-40a6-ab09-8f27213a7cbc\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.569731 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.569593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/295af638-35e1-40a6-ab09-8f27213a7cbc-tmp\") pod \"opendatahub-operator-controller-manager-694bbb96f5-z5mv2\" (UID: \"295af638-35e1-40a6-ab09-8f27213a7cbc\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.569731 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.569696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/295af638-35e1-40a6-ab09-8f27213a7cbc-manifests\") pod \"opendatahub-operator-controller-manager-694bbb96f5-z5mv2\" (UID: \"295af638-35e1-40a6-ab09-8f27213a7cbc\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.569844 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.569733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/295af638-35e1-40a6-ab09-8f27213a7cbc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694bbb96f5-z5mv2\" (UID: \"295af638-35e1-40a6-ab09-8f27213a7cbc\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.569844 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.569765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/295af638-35e1-40a6-ab09-8f27213a7cbc-webhook-cert\") pod \"opendatahub-operator-controller-manager-694bbb96f5-z5mv2\" (UID: \"295af638-35e1-40a6-ab09-8f27213a7cbc\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.670989 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.670960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/295af638-35e1-40a6-ab09-8f27213a7cbc-manifests\") pod \"opendatahub-operator-controller-manager-694bbb96f5-z5mv2\" (UID: \"295af638-35e1-40a6-ab09-8f27213a7cbc\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.671113 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.670997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/295af638-35e1-40a6-ab09-8f27213a7cbc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694bbb96f5-z5mv2\" (UID: \"295af638-35e1-40a6-ab09-8f27213a7cbc\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.671113 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.671023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/295af638-35e1-40a6-ab09-8f27213a7cbc-webhook-cert\") pod \"opendatahub-operator-controller-manager-694bbb96f5-z5mv2\" (UID: \"295af638-35e1-40a6-ab09-8f27213a7cbc\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.671113 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.671060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjbhp\" (UniqueName: \"kubernetes.io/projected/295af638-35e1-40a6-ab09-8f27213a7cbc-kube-api-access-fjbhp\") pod \"opendatahub-operator-controller-manager-694bbb96f5-z5mv2\" (UID: \"295af638-35e1-40a6-ab09-8f27213a7cbc\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.671113 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.671098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/295af638-35e1-40a6-ab09-8f27213a7cbc-tmp\") pod \"opendatahub-operator-controller-manager-694bbb96f5-z5mv2\" (UID: \"295af638-35e1-40a6-ab09-8f27213a7cbc\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.671351 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.671329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/295af638-35e1-40a6-ab09-8f27213a7cbc-manifests\") pod \"opendatahub-operator-controller-manager-694bbb96f5-z5mv2\" (UID: \"295af638-35e1-40a6-ab09-8f27213a7cbc\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.673264 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.673237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/295af638-35e1-40a6-ab09-8f27213a7cbc-tmp\") pod \"opendatahub-operator-controller-manager-694bbb96f5-z5mv2\" (UID: \"295af638-35e1-40a6-ab09-8f27213a7cbc\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.673678 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.673657    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/295af638-35e1-40a6-ab09-8f27213a7cbc-webhook-cert\") pod \"opendatahub-operator-controller-manager-694bbb96f5-z5mv2\" (UID: \"295af638-35e1-40a6-ab09-8f27213a7cbc\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.673755 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.673657    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/295af638-35e1-40a6-ab09-8f27213a7cbc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694bbb96f5-z5mv2\" (UID: \"295af638-35e1-40a6-ab09-8f27213a7cbc\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.684623 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.684589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjbhp\" (UniqueName: \"kubernetes.io/projected/295af638-35e1-40a6-ab09-8f27213a7cbc-kube-api-access-fjbhp\") pod \"opendatahub-operator-controller-manager-694bbb96f5-z5mv2\" (UID: \"295af638-35e1-40a6-ab09-8f27213a7cbc\") " pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:42.811653 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:42.811572    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:43.095988 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:43.095965    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"]
Sep 12 00:48:43.098346 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:48:43.098315    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295af638_35e1_40a6_ab09_8f27213a7cbc.slice/crio-539da11e04a164272473bece29e00594703a3b2bd108c121ab49f774eff87218 WatchSource:0}: Error finding container 539da11e04a164272473bece29e00594703a3b2bd108c121ab49f774eff87218: Status 404 returned error can't find the container with id 539da11e04a164272473bece29e00594703a3b2bd108c121ab49f774eff87218
Sep 12 00:48:43.758053 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:43.758012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2" event={"ID":"295af638-35e1-40a6-ab09-8f27213a7cbc","Type":"ContainerStarted","Data":"539da11e04a164272473bece29e00594703a3b2bd108c121ab49f774eff87218"}
Sep 12 00:48:46.770049 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:46.770019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2" event={"ID":"295af638-35e1-40a6-ab09-8f27213a7cbc","Type":"ContainerStarted","Data":"44b047a63ef87bafe1dede9b98b1f354c487750694dfb1753581cdfbddd058c5"}
Sep 12 00:48:47.774525 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:47.774485    2579 generic.go:356] "Generic (PLEG): container finished" podID="295af638-35e1-40a6-ab09-8f27213a7cbc" containerID="44b047a63ef87bafe1dede9b98b1f354c487750694dfb1753581cdfbddd058c5" exitCode=0
Sep 12 00:48:47.774895 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:47.774533    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2" event={"ID":"295af638-35e1-40a6-ab09-8f27213a7cbc","Type":"ContainerDied","Data":"44b047a63ef87bafe1dede9b98b1f354c487750694dfb1753581cdfbddd058c5"}
Sep 12 00:48:48.779846 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:48.779809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2" event={"ID":"295af638-35e1-40a6-ab09-8f27213a7cbc","Type":"ContainerStarted","Data":"218f2344b8e9a0ad7bac5f567ac2150b2f07ca6c0e0cb61a410644317cc943b2"}
Sep 12 00:48:48.780193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:48.779916    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:48:48.805384 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:48.805338    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2" podStartSLOduration=2.6234341949999997 podStartE2EDuration="6.805324444s" totalImagesPullingTime="3.124972722s" totalInitContainerRuntime="1.056917527s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:48:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:48:48.804082178 +0000 UTC m=+353.723373975" watchObservedRunningTime="2026-09-12 00:48:48.805324444 +0000 UTC m=+353.724616230"
Sep 12 00:48:59.785826 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:48:59.785798    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-694bbb96f5-z5mv2"
Sep 12 00:49:02.338253 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.338217    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6"]
Sep 12 00:49:02.346405 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.346379    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6"
Sep 12 00:49:02.349738 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.349719    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 12 00:49:02.349851 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.349719    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 12 00:49:02.350000 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.349986    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 12 00:49:02.350724 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.350707    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2gsws\""
Sep 12 00:49:02.350724 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.350707    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 12 00:49:02.358583 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.358561    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6"]
Sep 12 00:49:02.511890 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.511862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2dd22ae-e545-4441-a1e0-1a1d1dc46079-tmp\") pod \"kube-auth-proxy-5f75b6b767-wrxh6\" (UID: \"d2dd22ae-e545-4441-a1e0-1a1d1dc46079\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6"
Sep 12 00:49:02.512013 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.511901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2dd22ae-e545-4441-a1e0-1a1d1dc46079-tls-certs\") pod \"kube-auth-proxy-5f75b6b767-wrxh6\" (UID: \"d2dd22ae-e545-4441-a1e0-1a1d1dc46079\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6"
Sep 12 00:49:02.512013 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.511919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd7xb\" (UniqueName: \"kubernetes.io/projected/d2dd22ae-e545-4441-a1e0-1a1d1dc46079-kube-api-access-wd7xb\") pod \"kube-auth-proxy-5f75b6b767-wrxh6\" (UID: \"d2dd22ae-e545-4441-a1e0-1a1d1dc46079\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6"
Sep 12 00:49:02.612649 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.612576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2dd22ae-e545-4441-a1e0-1a1d1dc46079-tls-certs\") pod \"kube-auth-proxy-5f75b6b767-wrxh6\" (UID: \"d2dd22ae-e545-4441-a1e0-1a1d1dc46079\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6"
Sep 12 00:49:02.612649 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.612619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd7xb\" (UniqueName: \"kubernetes.io/projected/d2dd22ae-e545-4441-a1e0-1a1d1dc46079-kube-api-access-wd7xb\") pod \"kube-auth-proxy-5f75b6b767-wrxh6\" (UID: \"d2dd22ae-e545-4441-a1e0-1a1d1dc46079\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6"
Sep 12 00:49:02.612767 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.612663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2dd22ae-e545-4441-a1e0-1a1d1dc46079-tmp\") pod \"kube-auth-proxy-5f75b6b767-wrxh6\" (UID: \"d2dd22ae-e545-4441-a1e0-1a1d1dc46079\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6"
Sep 12 00:49:02.614822 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.614793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2dd22ae-e545-4441-a1e0-1a1d1dc46079-tmp\") pod \"kube-auth-proxy-5f75b6b767-wrxh6\" (UID: \"d2dd22ae-e545-4441-a1e0-1a1d1dc46079\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6"
Sep 12 00:49:02.614998 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.614977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2dd22ae-e545-4441-a1e0-1a1d1dc46079-tls-certs\") pod \"kube-auth-proxy-5f75b6b767-wrxh6\" (UID: \"d2dd22ae-e545-4441-a1e0-1a1d1dc46079\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6"
Sep 12 00:49:02.662911 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.662882    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd7xb\" (UniqueName: \"kubernetes.io/projected/d2dd22ae-e545-4441-a1e0-1a1d1dc46079-kube-api-access-wd7xb\") pod \"kube-auth-proxy-5f75b6b767-wrxh6\" (UID: \"d2dd22ae-e545-4441-a1e0-1a1d1dc46079\") " pod="openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6"
Sep 12 00:49:02.955102 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:02.955068    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6"
Sep 12 00:49:03.233444 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:03.233370    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6"]
Sep 12 00:49:03.236532 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:49:03.236504    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2dd22ae_e545_4441_a1e0_1a1d1dc46079.slice/crio-df8e96803d3d26725822df3142922aa001f6fa52ba5acf265922486795d5ea3c WatchSource:0}: Error finding container df8e96803d3d26725822df3142922aa001f6fa52ba5acf265922486795d5ea3c: Status 404 returned error can't find the container with id df8e96803d3d26725822df3142922aa001f6fa52ba5acf265922486795d5ea3c
Sep 12 00:49:03.834423 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:03.834347    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6" event={"ID":"d2dd22ae-e545-4441-a1e0-1a1d1dc46079","Type":"ContainerStarted","Data":"df8e96803d3d26725822df3142922aa001f6fa52ba5acf265922486795d5ea3c"}
Sep 12 00:49:03.900794 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:03.900759    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"]
Sep 12 00:49:03.904272 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:03.904236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:03.906987 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:03.906965    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 12 00:49:03.907593 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:03.907147    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sjm6q\""
Sep 12 00:49:03.907593 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:03.907280    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 12 00:49:03.907593 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:03.907353    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 12 00:49:03.907593 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:03.907468    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 12 00:49:03.929199 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:03.929144    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"]
Sep 12 00:49:04.024561 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.024525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.024764 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.024573    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7d16364-53c1-4945-ab07-f0d6042bb173-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.024764 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.024629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.024764 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.024719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtv5f\" (UniqueName: \"kubernetes.io/projected/c7d16364-53c1-4945-ab07-f0d6042bb173-kube-api-access-wtv5f\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.024923 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.024803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.024923 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.024836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.024923 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.024864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.125885 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.125800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.125885 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.125840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.125885 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.125865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.126157 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.125922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.126157 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.125944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7d16364-53c1-4945-ab07-f0d6042bb173-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.126157 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.125990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.126157 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.126019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtv5f\" (UniqueName: \"kubernetes.io/projected/c7d16364-53c1-4945-ab07-f0d6042bb173-kube-api-access-wtv5f\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.126711 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.126650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.128237 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.128214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.128345 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.128316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.128622 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.128578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7d16364-53c1-4945-ab07-f0d6042bb173-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.128693 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.128672    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.134922 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.134893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.134922 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.134901    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtv5f\" (UniqueName: \"kubernetes.io/projected/c7d16364-53c1-4945-ab07-f0d6042bb173-kube-api-access-wtv5f\") pod \"istiod-openshift-gateway-7b7995d5cd-b6prj\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.216982 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.216948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:04.504384 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:04.504305    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"]
Sep 12 00:49:05.649904 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:49:05.649868    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d16364_53c1_4945_ab07_f0d6042bb173.slice/crio-dcb32db5910fd88c7c050a42a4efcc7a0e2f32031bcf238e4c45ce5f6f3a330d WatchSource:0}: Error finding container dcb32db5910fd88c7c050a42a4efcc7a0e2f32031bcf238e4c45ce5f6f3a330d: Status 404 returned error can't find the container with id dcb32db5910fd88c7c050a42a4efcc7a0e2f32031bcf238e4c45ce5f6f3a330d
Sep 12 00:49:05.842682 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:05.842645    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj" event={"ID":"c7d16364-53c1-4945-ab07-f0d6042bb173","Type":"ContainerStarted","Data":"dcb32db5910fd88c7c050a42a4efcc7a0e2f32031bcf238e4c45ce5f6f3a330d"}
Sep 12 00:49:05.844143 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:05.844116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6" event={"ID":"d2dd22ae-e545-4441-a1e0-1a1d1dc46079","Type":"ContainerStarted","Data":"07985f4f664e7b85aa54ed0ae9ef2288179dd31ccba080f4d0ef6827dbad1a72"}
Sep 12 00:49:05.862487 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:05.862442    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5f75b6b767-wrxh6" podStartSLOduration=1.399298399 podStartE2EDuration="3.862427875s" totalImagesPullingTime="2.463129476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:49:05.861552679 +0000 UTC m=+370.780844467" watchObservedRunningTime="2026-09-12 00:49:05.862427875 +0000 UTC m=+370.781719661"
Sep 12 00:49:08.299331 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:08.299298    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:49:08.299574 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:08.299361    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:49:08.859338 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:08.859299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj" event={"ID":"c7d16364-53c1-4945-ab07-f0d6042bb173","Type":"ContainerStarted","Data":"5800dc53c4c6d2ed5cc440c8824a69ce96160b5a5b0cdc9d6c3c8e6a74c351fb"}
Sep 12 00:49:08.859520 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:08.859375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:08.890782 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:08.890724    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj" podStartSLOduration=3.243599435 podStartE2EDuration="5.890705384s" totalImagesPullingTime="2.647105949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:49:08.889524025 +0000 UTC m=+373.808815813" watchObservedRunningTime="2026-09-12 00:49:08.890705384 +0000 UTC m=+373.809997173"
Sep 12 00:49:09.220914 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.220875    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"]
Sep 12 00:49:09.224039 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.224018    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.226777 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.226751    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-p5n7s\""
Sep 12 00:49:09.244617 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.242127    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"]
Sep 12 00:49:09.369122 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.369092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.369502 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.369131    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.369502 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.369169    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgnss\" (UniqueName: \"kubernetes.io/projected/6c56d294-39ad-4730-873d-e8ba723ee057-kube-api-access-jgnss\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.369502 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.369260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.369502 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.369288    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.369502 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.369308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c56d294-39ad-4730-873d-e8ba723ee057-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.369502 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.369333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.369502 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.369353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c56d294-39ad-4730-873d-e8ba723ee057-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.369502 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.369370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c56d294-39ad-4730-873d-e8ba723ee057-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.469931 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.469896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.469931 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.469928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.470151 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.469960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgnss\" (UniqueName: \"kubernetes.io/projected/6c56d294-39ad-4730-873d-e8ba723ee057-kube-api-access-jgnss\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.470151 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.470082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.470151 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.470111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.470151 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.470140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c56d294-39ad-4730-873d-e8ba723ee057-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.470350 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.470164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.470350 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.470188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c56d294-39ad-4730-873d-e8ba723ee057-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.470350 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.470213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c56d294-39ad-4730-873d-e8ba723ee057-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.470495 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.470406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.470629 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.470557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.470759 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.470674    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.470759 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.470718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.470948 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.470906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c56d294-39ad-4730-873d-e8ba723ee057-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.472480 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.472458    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.472574 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.472553    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c56d294-39ad-4730-873d-e8ba723ee057-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.478447 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.478423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgnss\" (UniqueName: \"kubernetes.io/projected/6c56d294-39ad-4730-873d-e8ba723ee057-kube-api-access-jgnss\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.479137 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.479118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c56d294-39ad-4730-873d-e8ba723ee057-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcbncx4\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.533627 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.533590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:09.807512 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.807488    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"]
Sep 12 00:49:09.809753 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:49:09.809725    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c56d294_39ad_4730_873d_e8ba723ee057.slice/crio-f1cde89b1c81259afc13c2107c6c0c085545bc2318b5528cd97c7c58d0b9b9a8 WatchSource:0}: Error finding container f1cde89b1c81259afc13c2107c6c0c085545bc2318b5528cd97c7c58d0b9b9a8: Status 404 returned error can't find the container with id f1cde89b1c81259afc13c2107c6c0c085545bc2318b5528cd97c7c58d0b9b9a8
Sep 12 00:49:09.864210 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.864183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" event={"ID":"6c56d294-39ad-4730-873d-e8ba723ee057","Type":"ContainerStarted","Data":"f1cde89b1c81259afc13c2107c6c0c085545bc2318b5528cd97c7c58d0b9b9a8"}
Sep 12 00:49:09.865893 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:09.865874    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:12.047780 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:12.047749    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:49:12.048018 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:12.047814    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:49:12.048018 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:12.047869    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:49:12.877383 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:12.877355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" event={"ID":"6c56d294-39ad-4730-873d-e8ba723ee057","Type":"ContainerStarted","Data":"48afe4d36ebdba6a665fbd32f824255a0e19aa7392b7f61ac9b5636630f9ed7c"}
Sep 12 00:49:12.913261 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:12.913216    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" podStartSLOduration=1.677482997 podStartE2EDuration="3.913200129s" totalImagesPullingTime="2.235717132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:49:12.911546717 +0000 UTC m=+377.830838504" watchObservedRunningTime="2026-09-12 00:49:12.913200129 +0000 UTC m=+377.832491915"
Sep 12 00:49:13.534617 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:13.534573    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:13.535937 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:13.535913    2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused" start-of-body=
Sep 12 00:49:13.536038 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:13.535966    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" podUID="6c56d294-39ad-4730-873d-e8ba723ee057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused"
Sep 12 00:49:14.534742 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:14.534710    2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused" start-of-body=
Sep 12 00:49:14.535095 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:14.534762    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" podUID="6c56d294-39ad-4730-873d-e8ba723ee057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused"
Sep 12 00:49:15.533964 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:15.533929    2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused" start-of-body=
Sep 12 00:49:15.534138 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:15.533981    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" podUID="6c56d294-39ad-4730-873d-e8ba723ee057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused"
Sep 12 00:49:16.534915 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:16.534878    2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused" start-of-body=
Sep 12 00:49:16.535274 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:16.534937    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" podUID="6c56d294-39ad-4730-873d-e8ba723ee057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused"
Sep 12 00:49:17.534672 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:17.534639    2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused" start-of-body=
Sep 12 00:49:17.534856 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:17.534698    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" podUID="6c56d294-39ad-4730-873d-e8ba723ee057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused"
Sep 12 00:49:18.534132 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:18.534092    2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused" start-of-body=
Sep 12 00:49:18.534570 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:18.534158    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" podUID="6c56d294-39ad-4730-873d-e8ba723ee057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused"
Sep 12 00:49:19.534033 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.533997    2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused" start-of-body=
Sep 12 00:49:19.534218 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.534059    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" podUID="6c56d294-39ad-4730-873d-e8ba723ee057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused"
Sep 12 00:49:19.596828 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.596797    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"]
Sep 12 00:49:19.670997 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.670971    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"]
Sep 12 00:49:19.671126 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.671051    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.736152 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.736124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55eebaf5-c290-42c2-90ce-dd5934366f6c-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.736271 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.736159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/55eebaf5-c290-42c2-90ce-dd5934366f6c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.736271 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.736187    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjh5c\" (UniqueName: \"kubernetes.io/projected/55eebaf5-c290-42c2-90ce-dd5934366f6c-kube-api-access-gjh5c\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.736271 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.736257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55eebaf5-c290-42c2-90ce-dd5934366f6c-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.736375 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.736297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/55eebaf5-c290-42c2-90ce-dd5934366f6c-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.736375 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.736315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/55eebaf5-c290-42c2-90ce-dd5934366f6c-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.736375 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.736331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/55eebaf5-c290-42c2-90ce-dd5934366f6c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.836924 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.836861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjh5c\" (UniqueName: \"kubernetes.io/projected/55eebaf5-c290-42c2-90ce-dd5934366f6c-kube-api-access-gjh5c\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.836924 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.836898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55eebaf5-c290-42c2-90ce-dd5934366f6c-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.836924 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.836923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/55eebaf5-c290-42c2-90ce-dd5934366f6c-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.837145 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.836941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/55eebaf5-c290-42c2-90ce-dd5934366f6c-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.837145 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.836956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/55eebaf5-c290-42c2-90ce-dd5934366f6c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.837145 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.836992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55eebaf5-c290-42c2-90ce-dd5934366f6c-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.837145 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.837006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/55eebaf5-c290-42c2-90ce-dd5934366f6c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.837861 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.837834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/55eebaf5-c290-42c2-90ce-dd5934366f6c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.839338 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.839308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/55eebaf5-c290-42c2-90ce-dd5934366f6c-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.839438 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.839351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/55eebaf5-c290-42c2-90ce-dd5934366f6c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.839438 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.839430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55eebaf5-c290-42c2-90ce-dd5934366f6c-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.839548 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.839494    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/55eebaf5-c290-42c2-90ce-dd5934366f6c-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.850753 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.850730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55eebaf5-c290-42c2-90ce-dd5934366f6c-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.850858 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.850841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjh5c\" (UniqueName: \"kubernetes.io/projected/55eebaf5-c290-42c2-90ce-dd5934366f6c-kube-api-access-gjh5c\") pod \"istiod-openshift-gateway-5677ccb49b-47vv5\" (UID: \"55eebaf5-c290-42c2-90ce-dd5934366f6c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:19.979055 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:19.979034    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:20.269542 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:20.269519    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"]
Sep 12 00:49:20.271828 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:49:20.271803    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55eebaf5_c290_42c2_90ce_dd5934366f6c.slice/crio-91074a45f3b79c521fd567c32059bb75617b2d976b7052bb730425c36a646d1a WatchSource:0}: Error finding container 91074a45f3b79c521fd567c32059bb75617b2d976b7052bb730425c36a646d1a: Status 404 returned error can't find the container with id 91074a45f3b79c521fd567c32059bb75617b2d976b7052bb730425c36a646d1a
Sep 12 00:49:20.274794 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:20.274762    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:49:20.274884 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:20.274842    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:49:20.534972 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:20.534892    2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused" start-of-body=
Sep 12 00:49:20.534972 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:20.534947    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" podUID="6c56d294-39ad-4730-873d-e8ba723ee057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused"
Sep 12 00:49:20.909099 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:20.909010    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5" event={"ID":"55eebaf5-c290-42c2-90ce-dd5934366f6c","Type":"ContainerStarted","Data":"0b898d7a18dd750dc976a1fdfda84fc433331eceb8a41b533ed7a3091ab0f8ea"}
Sep 12 00:49:20.909247 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:20.909152    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:20.909247 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:20.909200    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5" event={"ID":"55eebaf5-c290-42c2-90ce-dd5934366f6c","Type":"ContainerStarted","Data":"91074a45f3b79c521fd567c32059bb75617b2d976b7052bb730425c36a646d1a"}
Sep 12 00:49:20.957026 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:20.956980    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5" podStartSLOduration=1.9569624669999999 podStartE2EDuration="1.956962467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:49:20.955747924 +0000 UTC m=+385.875039711" watchObservedRunningTime="2026-09-12 00:49:20.956962467 +0000 UTC m=+385.876254252"
Sep 12 00:49:21.534244 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:21.534210    2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused" start-of-body=
Sep 12 00:49:21.534402 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:21.534262    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" podUID="6c56d294-39ad-4730-873d-e8ba723ee057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused"
Sep 12 00:49:21.914507 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:21.914480    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-47vv5"
Sep 12 00:49:21.994251 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:21.994219    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"]
Sep 12 00:49:21.994416 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:21.994395    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj" podUID="c7d16364-53c1-4945-ab07-f0d6042bb173" containerName="discovery" containerID="cri-o://5800dc53c4c6d2ed5cc440c8824a69ce96160b5a5b0cdc9d6c3c8e6a74c351fb" gracePeriod=30
Sep 12 00:49:22.218681 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.218658    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:22.254677 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.254651    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-token\") pod \"c7d16364-53c1-4945-ab07-f0d6042bb173\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") "
Sep 12 00:49:22.254806 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.254714    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-csr-dns-cert\") pod \"c7d16364-53c1-4945-ab07-f0d6042bb173\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") "
Sep 12 00:49:22.254806 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.254767    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-kubeconfig\") pod \"c7d16364-53c1-4945-ab07-f0d6042bb173\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") "
Sep 12 00:49:22.254806 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.254799    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-cacerts\") pod \"c7d16364-53c1-4945-ab07-f0d6042bb173\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") "
Sep 12 00:49:22.254806 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.254815    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtv5f\" (UniqueName: \"kubernetes.io/projected/c7d16364-53c1-4945-ab07-f0d6042bb173-kube-api-access-wtv5f\") pod \"c7d16364-53c1-4945-ab07-f0d6042bb173\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") "
Sep 12 00:49:22.255053 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.254857    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7d16364-53c1-4945-ab07-f0d6042bb173-local-certs\") pod \"c7d16364-53c1-4945-ab07-f0d6042bb173\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") "
Sep 12 00:49:22.255053 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.254892    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-csr-ca-configmap\") pod \"c7d16364-53c1-4945-ab07-f0d6042bb173\" (UID: \"c7d16364-53c1-4945-ab07-f0d6042bb173\") "
Sep 12 00:49:22.255659 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.255396    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "c7d16364-53c1-4945-ab07-f0d6042bb173" (UID: "c7d16364-53c1-4945-ab07-f0d6042bb173"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:49:22.257551 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.257519    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "c7d16364-53c1-4945-ab07-f0d6042bb173" (UID: "c7d16364-53c1-4945-ab07-f0d6042bb173"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:49:22.257712 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.257686    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-cacerts" (OuterVolumeSpecName: "cacerts") pod "c7d16364-53c1-4945-ab07-f0d6042bb173" (UID: "c7d16364-53c1-4945-ab07-f0d6042bb173"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:49:22.258271 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.258235    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d16364-53c1-4945-ab07-f0d6042bb173-local-certs" (OuterVolumeSpecName: "local-certs") pod "c7d16364-53c1-4945-ab07-f0d6042bb173" (UID: "c7d16364-53c1-4945-ab07-f0d6042bb173"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:22.258271 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.258248    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d16364-53c1-4945-ab07-f0d6042bb173-kube-api-access-wtv5f" (OuterVolumeSpecName: "kube-api-access-wtv5f") pod "c7d16364-53c1-4945-ab07-f0d6042bb173" (UID: "c7d16364-53c1-4945-ab07-f0d6042bb173"). InnerVolumeSpecName "kube-api-access-wtv5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:49:22.258420 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.258314    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-token" (OuterVolumeSpecName: "istio-token") pod "c7d16364-53c1-4945-ab07-f0d6042bb173" (UID: "c7d16364-53c1-4945-ab07-f0d6042bb173"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:49:22.258869 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.258835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "c7d16364-53c1-4945-ab07-f0d6042bb173" (UID: "c7d16364-53c1-4945-ab07-f0d6042bb173"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 00:49:22.356362 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.356334    2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-csr-dns-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:22.356362 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.356356    2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-kubeconfig\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:22.356362 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.356366    2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7d16364-53c1-4945-ab07-f0d6042bb173-cacerts\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:22.356526 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.356375    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtv5f\" (UniqueName: \"kubernetes.io/projected/c7d16364-53c1-4945-ab07-f0d6042bb173-kube-api-access-wtv5f\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:22.356526 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.356385    2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7d16364-53c1-4945-ab07-f0d6042bb173-local-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:22.356526 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.356392    2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-csr-ca-configmap\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:22.356526 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.356400    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7d16364-53c1-4945-ab07-f0d6042bb173-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:22.534242 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.534168    2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused" start-of-body=
Sep 12 00:49:22.534242 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.534218    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" podUID="6c56d294-39ad-4730-873d-e8ba723ee057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.17:15021/healthz/ready\": dial tcp 10.132.0.17:15021: connect: connection refused"
Sep 12 00:49:22.640200 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.640172    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m"]
Sep 12 00:49:22.640433 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.640422    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d16364-53c1-4945-ab07-f0d6042bb173" containerName="discovery"
Sep 12 00:49:22.640482 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.640436    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d16364-53c1-4945-ab07-f0d6042bb173" containerName="discovery"
Sep 12 00:49:22.640482 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.640476    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d16364-53c1-4945-ab07-f0d6042bb173" containerName="discovery"
Sep 12 00:49:22.642045 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.642029    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m"
Sep 12 00:49:22.646635 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.646593    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-x5hqt\""
Sep 12 00:49:22.647248 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.647227    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 12 00:49:22.658367 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.658343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5ef8ce8-3d5e-4344-b605-b782c357bf1c-cert\") pod \"kserve-controller-manager-d6f4b65b8-d4q4m\" (UID: \"d5ef8ce8-3d5e-4344-b605-b782c357bf1c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m"
Sep 12 00:49:22.658444 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.658395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v298p\" (UniqueName: \"kubernetes.io/projected/d5ef8ce8-3d5e-4344-b605-b782c357bf1c-kube-api-access-v298p\") pod \"kserve-controller-manager-d6f4b65b8-d4q4m\" (UID: \"d5ef8ce8-3d5e-4344-b605-b782c357bf1c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m"
Sep 12 00:49:22.677978 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.677954    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m"]
Sep 12 00:49:22.759226 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.759193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5ef8ce8-3d5e-4344-b605-b782c357bf1c-cert\") pod \"kserve-controller-manager-d6f4b65b8-d4q4m\" (UID: \"d5ef8ce8-3d5e-4344-b605-b782c357bf1c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m"
Sep 12 00:49:22.759383 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.759254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v298p\" (UniqueName: \"kubernetes.io/projected/d5ef8ce8-3d5e-4344-b605-b782c357bf1c-kube-api-access-v298p\") pod \"kserve-controller-manager-d6f4b65b8-d4q4m\" (UID: \"d5ef8ce8-3d5e-4344-b605-b782c357bf1c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m"
Sep 12 00:49:22.759383 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:49:22.759333    2579 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 12 00:49:22.759493 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:49:22.759387    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5ef8ce8-3d5e-4344-b605-b782c357bf1c-cert podName:d5ef8ce8-3d5e-4344-b605-b782c357bf1c nodeName:}" failed. No retries permitted until 2026-09-12 00:49:23.259373166 +0000 UTC m=+388.178664931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d5ef8ce8-3d5e-4344-b605-b782c357bf1c-cert") pod "kserve-controller-manager-d6f4b65b8-d4q4m" (UID: "d5ef8ce8-3d5e-4344-b605-b782c357bf1c") : secret "kserve-webhook-server-cert" not found
Sep 12 00:49:22.782575 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.782546    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v298p\" (UniqueName: \"kubernetes.io/projected/d5ef8ce8-3d5e-4344-b605-b782c357bf1c-kube-api-access-v298p\") pod \"kserve-controller-manager-d6f4b65b8-d4q4m\" (UID: \"d5ef8ce8-3d5e-4344-b605-b782c357bf1c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m"
Sep 12 00:49:22.917683 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.917655    2579 generic.go:356] "Generic (PLEG): container finished" podID="c7d16364-53c1-4945-ab07-f0d6042bb173" containerID="5800dc53c4c6d2ed5cc440c8824a69ce96160b5a5b0cdc9d6c3c8e6a74c351fb" exitCode=0
Sep 12 00:49:22.918042 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.917698    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"
Sep 12 00:49:22.918042 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.917730    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj" event={"ID":"c7d16364-53c1-4945-ab07-f0d6042bb173","Type":"ContainerDied","Data":"5800dc53c4c6d2ed5cc440c8824a69ce96160b5a5b0cdc9d6c3c8e6a74c351fb"}
Sep 12 00:49:22.918042 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.917769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj" event={"ID":"c7d16364-53c1-4945-ab07-f0d6042bb173","Type":"ContainerDied","Data":"dcb32db5910fd88c7c050a42a4efcc7a0e2f32031bcf238e4c45ce5f6f3a330d"}
Sep 12 00:49:22.918042 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.917793    2579 scope.go:117] "RemoveContainer" containerID="5800dc53c4c6d2ed5cc440c8824a69ce96160b5a5b0cdc9d6c3c8e6a74c351fb"
Sep 12 00:49:22.926044 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.926026    2579 scope.go:117] "RemoveContainer" containerID="5800dc53c4c6d2ed5cc440c8824a69ce96160b5a5b0cdc9d6c3c8e6a74c351fb"
Sep 12 00:49:22.926306 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:49:22.926288    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5800dc53c4c6d2ed5cc440c8824a69ce96160b5a5b0cdc9d6c3c8e6a74c351fb\": container with ID starting with 5800dc53c4c6d2ed5cc440c8824a69ce96160b5a5b0cdc9d6c3c8e6a74c351fb not found: ID does not exist" containerID="5800dc53c4c6d2ed5cc440c8824a69ce96160b5a5b0cdc9d6c3c8e6a74c351fb"
Sep 12 00:49:22.926369 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.926312    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5800dc53c4c6d2ed5cc440c8824a69ce96160b5a5b0cdc9d6c3c8e6a74c351fb"} err="failed to get container status \"5800dc53c4c6d2ed5cc440c8824a69ce96160b5a5b0cdc9d6c3c8e6a74c351fb\": rpc error: code = NotFound desc = could not find container \"5800dc53c4c6d2ed5cc440c8824a69ce96160b5a5b0cdc9d6c3c8e6a74c351fb\": container with ID starting with 5800dc53c4c6d2ed5cc440c8824a69ce96160b5a5b0cdc9d6c3c8e6a74c351fb not found: ID does not exist"
Sep 12 00:49:22.944078 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.944058    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"]
Sep 12 00:49:22.949265 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:22.949245    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-b6prj"]
Sep 12 00:49:23.263443 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:23.263371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5ef8ce8-3d5e-4344-b605-b782c357bf1c-cert\") pod \"kserve-controller-manager-d6f4b65b8-d4q4m\" (UID: \"d5ef8ce8-3d5e-4344-b605-b782c357bf1c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m"
Sep 12 00:49:23.265664 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:23.265644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d5ef8ce8-3d5e-4344-b605-b782c357bf1c-cert\") pod \"kserve-controller-manager-d6f4b65b8-d4q4m\" (UID: \"d5ef8ce8-3d5e-4344-b605-b782c357bf1c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m"
Sep 12 00:49:23.472727 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:23.472694    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"]
Sep 12 00:49:23.472909 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:23.472885    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" podUID="6c56d294-39ad-4730-873d-e8ba723ee057" containerName="istio-proxy" containerID="cri-o://48afe4d36ebdba6a665fbd32f824255a0e19aa7392b7f61ac9b5636630f9ed7c" gracePeriod=30
Sep 12 00:49:23.552201 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:23.552133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m"
Sep 12 00:49:23.590945 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:23.590915    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d16364-53c1-4945-ab07-f0d6042bb173" path="/var/lib/kubelet/pods/c7d16364-53c1-4945-ab07-f0d6042bb173/volumes"
Sep 12 00:49:23.825267 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:23.825242    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m"]
Sep 12 00:49:23.826670 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:49:23.826645    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ef8ce8_3d5e_4344_b605_b782c357bf1c.slice/crio-53e91ad5f97000729b3f1f4193e91b6cf40defac39c0164535aa3138546d171c WatchSource:0}: Error finding container 53e91ad5f97000729b3f1f4193e91b6cf40defac39c0164535aa3138546d171c: Status 404 returned error can't find the container with id 53e91ad5f97000729b3f1f4193e91b6cf40defac39c0164535aa3138546d171c
Sep 12 00:49:23.921863 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:23.921828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m" event={"ID":"d5ef8ce8-3d5e-4344-b605-b782c357bf1c","Type":"ContainerStarted","Data":"53e91ad5f97000729b3f1f4193e91b6cf40defac39c0164535aa3138546d171c"}
Sep 12 00:49:24.534781 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:24.534744    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:25.934797 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:25.934760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m" event={"ID":"d5ef8ce8-3d5e-4344-b605-b782c357bf1c","Type":"ContainerStarted","Data":"310b913aeda6b0ea6ac01ea0f274e594d356f0705ebe302e46ea937ee3ea948c"}
Sep 12 00:49:25.935231 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:25.934894    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m"
Sep 12 00:49:25.976915 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:25.976871    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m" podStartSLOduration=2.035316003 podStartE2EDuration="3.976856623s" totalImagesPullingTime="1.94154062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:49:25.975826878 +0000 UTC m=+390.895118666" watchObservedRunningTime="2026-09-12 00:49:25.976856623 +0000 UTC m=+390.896148410"
Sep 12 00:49:28.713081 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.713058    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:28.802668 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.802574    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-istio-data\") pod \"6c56d294-39ad-4730-873d-e8ba723ee057\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") "
Sep 12 00:49:28.802668 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.802623    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-istio-envoy\") pod \"6c56d294-39ad-4730-873d-e8ba723ee057\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") "
Sep 12 00:49:28.802668 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.802649    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c56d294-39ad-4730-873d-e8ba723ee057-istio-podinfo\") pod \"6c56d294-39ad-4730-873d-e8ba723ee057\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") "
Sep 12 00:49:28.802668 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.802670    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-credential-socket\") pod \"6c56d294-39ad-4730-873d-e8ba723ee057\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") "
Sep 12 00:49:28.802984 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.802688    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-workload-socket\") pod \"6c56d294-39ad-4730-873d-e8ba723ee057\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") "
Sep 12 00:49:28.802984 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.802730    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgnss\" (UniqueName: \"kubernetes.io/projected/6c56d294-39ad-4730-873d-e8ba723ee057-kube-api-access-jgnss\") pod \"6c56d294-39ad-4730-873d-e8ba723ee057\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") "
Sep 12 00:49:28.802984 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.802823    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c56d294-39ad-4730-873d-e8ba723ee057-istiod-ca-cert\") pod \"6c56d294-39ad-4730-873d-e8ba723ee057\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") "
Sep 12 00:49:28.802984 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.802840    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c56d294-39ad-4730-873d-e8ba723ee057" (UID: "6c56d294-39ad-4730-873d-e8ba723ee057"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:28.802984 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.802874    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-workload-certs\") pod \"6c56d294-39ad-4730-873d-e8ba723ee057\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") "
Sep 12 00:49:28.802984 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.802899    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c56d294-39ad-4730-873d-e8ba723ee057-istio-token\") pod \"6c56d294-39ad-4730-873d-e8ba723ee057\" (UID: \"6c56d294-39ad-4730-873d-e8ba723ee057\") "
Sep 12 00:49:28.803274 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.803013    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c56d294-39ad-4730-873d-e8ba723ee057" (UID: "6c56d294-39ad-4730-873d-e8ba723ee057"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:28.803274 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.803025    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c56d294-39ad-4730-873d-e8ba723ee057" (UID: "6c56d294-39ad-4730-873d-e8ba723ee057"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:28.803274 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.803152    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:28.803274 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.803178    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:28.803274 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.803194    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:28.803274 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.803192    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c56d294-39ad-4730-873d-e8ba723ee057" (UID: "6c56d294-39ad-4730-873d-e8ba723ee057"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:28.803274 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.803229    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c56d294-39ad-4730-873d-e8ba723ee057-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c56d294-39ad-4730-873d-e8ba723ee057" (UID: "6c56d294-39ad-4730-873d-e8ba723ee057"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:49:28.804850 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.804823    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c56d294-39ad-4730-873d-e8ba723ee057" (UID: "6c56d294-39ad-4730-873d-e8ba723ee057"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:49:28.805020 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.805002    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c56d294-39ad-4730-873d-e8ba723ee057-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c56d294-39ad-4730-873d-e8ba723ee057" (UID: "6c56d294-39ad-4730-873d-e8ba723ee057"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 00:49:28.805312 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.805292    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c56d294-39ad-4730-873d-e8ba723ee057-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c56d294-39ad-4730-873d-e8ba723ee057" (UID: "6c56d294-39ad-4730-873d-e8ba723ee057"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:49:28.805371 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.805297    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c56d294-39ad-4730-873d-e8ba723ee057-kube-api-access-jgnss" (OuterVolumeSpecName: "kube-api-access-jgnss") pod "6c56d294-39ad-4730-873d-e8ba723ee057" (UID: "6c56d294-39ad-4730-873d-e8ba723ee057"). InnerVolumeSpecName "kube-api-access-jgnss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:49:28.903967 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.903945    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgnss\" (UniqueName: \"kubernetes.io/projected/6c56d294-39ad-4730-873d-e8ba723ee057-kube-api-access-jgnss\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:28.903967 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.903966    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c56d294-39ad-4730-873d-e8ba723ee057-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:28.904110 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.903975    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:28.904110 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.903983    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c56d294-39ad-4730-873d-e8ba723ee057-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:28.904110 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.903991    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c56d294-39ad-4730-873d-e8ba723ee057-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:28.904110 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.903998    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c56d294-39ad-4730-873d-e8ba723ee057-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:49:28.947385 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.947351    2579 generic.go:356] "Generic (PLEG): container finished" podID="6c56d294-39ad-4730-873d-e8ba723ee057" containerID="48afe4d36ebdba6a665fbd32f824255a0e19aa7392b7f61ac9b5636630f9ed7c" exitCode=0
Sep 12 00:49:28.947476 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.947400    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"
Sep 12 00:49:28.947476 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.947426    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" event={"ID":"6c56d294-39ad-4730-873d-e8ba723ee057","Type":"ContainerDied","Data":"48afe4d36ebdba6a665fbd32f824255a0e19aa7392b7f61ac9b5636630f9ed7c"}
Sep 12 00:49:28.947555 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.947480    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4" event={"ID":"6c56d294-39ad-4730-873d-e8ba723ee057","Type":"ContainerDied","Data":"f1cde89b1c81259afc13c2107c6c0c085545bc2318b5528cd97c7c58d0b9b9a8"}
Sep 12 00:49:28.947555 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.947509    2579 scope.go:117] "RemoveContainer" containerID="48afe4d36ebdba6a665fbd32f824255a0e19aa7392b7f61ac9b5636630f9ed7c"
Sep 12 00:49:28.955596 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.955578    2579 scope.go:117] "RemoveContainer" containerID="48afe4d36ebdba6a665fbd32f824255a0e19aa7392b7f61ac9b5636630f9ed7c"
Sep 12 00:49:28.955887 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:49:28.955868    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48afe4d36ebdba6a665fbd32f824255a0e19aa7392b7f61ac9b5636630f9ed7c\": container with ID starting with 48afe4d36ebdba6a665fbd32f824255a0e19aa7392b7f61ac9b5636630f9ed7c not found: ID does not exist" containerID="48afe4d36ebdba6a665fbd32f824255a0e19aa7392b7f61ac9b5636630f9ed7c"
Sep 12 00:49:28.955936 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.955896    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48afe4d36ebdba6a665fbd32f824255a0e19aa7392b7f61ac9b5636630f9ed7c"} err="failed to get container status \"48afe4d36ebdba6a665fbd32f824255a0e19aa7392b7f61ac9b5636630f9ed7c\": rpc error: code = NotFound desc = could not find container \"48afe4d36ebdba6a665fbd32f824255a0e19aa7392b7f61ac9b5636630f9ed7c\": container with ID starting with 48afe4d36ebdba6a665fbd32f824255a0e19aa7392b7f61ac9b5636630f9ed7c not found: ID does not exist"
Sep 12 00:49:28.969291 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.969268    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"]
Sep 12 00:49:28.973440 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:28.973416    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcbncx4"]
Sep 12 00:49:29.589363 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:29.589331    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c56d294-39ad-4730-873d-e8ba723ee057" path="/var/lib/kubelet/pods/6c56d294-39ad-4730-873d-e8ba723ee057/volumes"
Sep 12 00:49:56.944252 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:49:56.944223    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-d4q4m"
Sep 12 00:50:32.220258 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:32.220224    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5z5th"]
Sep 12 00:50:32.220828 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:32.220545    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c56d294-39ad-4730-873d-e8ba723ee057" containerName="istio-proxy"
Sep 12 00:50:32.220828 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:32.220563    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c56d294-39ad-4730-873d-e8ba723ee057" containerName="istio-proxy"
Sep 12 00:50:32.220828 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:32.220661    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c56d294-39ad-4730-873d-e8ba723ee057" containerName="istio-proxy"
Sep 12 00:50:32.223646 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:32.223629    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-5z5th"
Sep 12 00:50:32.226685 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:32.226665    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 12 00:50:32.226836 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:32.226815    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jc96d\""
Sep 12 00:50:32.226959 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:32.226941    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 12 00:50:32.236634 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:32.236616    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5z5th"]
Sep 12 00:50:32.339929 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:32.339899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f2c8\" (UniqueName: \"kubernetes.io/projected/b8e6b77d-6517-43bb-8c21-61296874691d-kube-api-access-4f2c8\") pod \"authorino-operator-76b478bd78-5z5th\" (UID: \"b8e6b77d-6517-43bb-8c21-61296874691d\") " pod="kuadrant-system/authorino-operator-76b478bd78-5z5th"
Sep 12 00:50:32.440797 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:32.440768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f2c8\" (UniqueName: \"kubernetes.io/projected/b8e6b77d-6517-43bb-8c21-61296874691d-kube-api-access-4f2c8\") pod \"authorino-operator-76b478bd78-5z5th\" (UID: \"b8e6b77d-6517-43bb-8c21-61296874691d\") " pod="kuadrant-system/authorino-operator-76b478bd78-5z5th"
Sep 12 00:50:32.452698 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:32.452674    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f2c8\" (UniqueName: \"kubernetes.io/projected/b8e6b77d-6517-43bb-8c21-61296874691d-kube-api-access-4f2c8\") pod \"authorino-operator-76b478bd78-5z5th\" (UID: \"b8e6b77d-6517-43bb-8c21-61296874691d\") " pod="kuadrant-system/authorino-operator-76b478bd78-5z5th"
Sep 12 00:50:32.532713 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:32.532656    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-5z5th"
Sep 12 00:50:32.813761 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:32.813736    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5z5th"]
Sep 12 00:50:32.815938 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:50:32.815915    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e6b77d_6517_43bb_8c21_61296874691d.slice/crio-310f5f1e43d038670c1d367df86f243308ae43690a94a795dc599cb80069330e WatchSource:0}: Error finding container 310f5f1e43d038670c1d367df86f243308ae43690a94a795dc599cb80069330e: Status 404 returned error can't find the container with id 310f5f1e43d038670c1d367df86f243308ae43690a94a795dc599cb80069330e
Sep 12 00:50:33.180753 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:33.180721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-5z5th" event={"ID":"b8e6b77d-6517-43bb-8c21-61296874691d","Type":"ContainerStarted","Data":"310f5f1e43d038670c1d367df86f243308ae43690a94a795dc599cb80069330e"}
Sep 12 00:50:36.195456 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:36.195422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-5z5th" event={"ID":"b8e6b77d-6517-43bb-8c21-61296874691d","Type":"ContainerStarted","Data":"c43eccf123b186ba93ea453f9070fd40c564bb18635a261ce8194fd17decece5"}
Sep 12 00:50:36.196002 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:36.195545    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-5z5th"
Sep 12 00:50:36.215882 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:36.215838    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-5z5th" podStartSLOduration=1.763032372 podStartE2EDuration="4.215825783s" totalImagesPullingTime="2.452793411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:50:36.214485798 +0000 UTC m=+461.133777605" watchObservedRunningTime="2026-09-12 00:50:36.215825783 +0000 UTC m=+461.135117570"
Sep 12 00:50:41.651819 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:41.651785    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"]
Sep 12 00:50:41.654971 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:41.654950    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"
Sep 12 00:50:41.657522 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:41.657503    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8nhgh\""
Sep 12 00:50:41.681632 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:41.681593    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"]
Sep 12 00:50:41.808872 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:41.808838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b01853b3-3cd5-4bdf-9c29-65cffe97c964-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bc4tj\" (UID: \"b01853b3-3cd5-4bdf-9c29-65cffe97c964\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"
Sep 12 00:50:41.809016 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:41.808919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjzrw\" (UniqueName: \"kubernetes.io/projected/b01853b3-3cd5-4bdf-9c29-65cffe97c964-kube-api-access-hjzrw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bc4tj\" (UID: \"b01853b3-3cd5-4bdf-9c29-65cffe97c964\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"
Sep 12 00:50:41.909397 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:41.909344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjzrw\" (UniqueName: \"kubernetes.io/projected/b01853b3-3cd5-4bdf-9c29-65cffe97c964-kube-api-access-hjzrw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bc4tj\" (UID: \"b01853b3-3cd5-4bdf-9c29-65cffe97c964\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"
Sep 12 00:50:41.909397 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:41.909382    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b01853b3-3cd5-4bdf-9c29-65cffe97c964-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bc4tj\" (UID: \"b01853b3-3cd5-4bdf-9c29-65cffe97c964\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"
Sep 12 00:50:41.909733 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:41.909716    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b01853b3-3cd5-4bdf-9c29-65cffe97c964-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bc4tj\" (UID: \"b01853b3-3cd5-4bdf-9c29-65cffe97c964\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"
Sep 12 00:50:41.923459 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:41.923442    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjzrw\" (UniqueName: \"kubernetes.io/projected/b01853b3-3cd5-4bdf-9c29-65cffe97c964-kube-api-access-hjzrw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bc4tj\" (UID: \"b01853b3-3cd5-4bdf-9c29-65cffe97c964\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"
Sep 12 00:50:41.964299 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:41.964271    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"
Sep 12 00:50:42.237485 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:42.237449    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"]
Sep 12 00:50:42.240647 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:50:42.240622    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01853b3_3cd5_4bdf_9c29_65cffe97c964.slice/crio-7f253e39ab44e9db57cdec93d3ed6019d11ff81ac38fdd79e2f2424efc8dd981 WatchSource:0}: Error finding container 7f253e39ab44e9db57cdec93d3ed6019d11ff81ac38fdd79e2f2424efc8dd981: Status 404 returned error can't find the container with id 7f253e39ab44e9db57cdec93d3ed6019d11ff81ac38fdd79e2f2424efc8dd981
Sep 12 00:50:43.225933 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:43.225889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj" event={"ID":"b01853b3-3cd5-4bdf-9c29-65cffe97c964","Type":"ContainerStarted","Data":"7f253e39ab44e9db57cdec93d3ed6019d11ff81ac38fdd79e2f2424efc8dd981"}
Sep 12 00:50:46.241293 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:46.241257    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj" event={"ID":"b01853b3-3cd5-4bdf-9c29-65cffe97c964","Type":"ContainerStarted","Data":"a63c6860e4bef5d8ea728611f876d0d2efbaa8b2b1c56c9f8bd7db4c182b0efd"}
Sep 12 00:50:46.241674 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:46.241368    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"
Sep 12 00:50:46.268167 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:46.268095    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj" podStartSLOduration=1.764392848 podStartE2EDuration="5.268081293s" totalImagesPullingTime="3.503688445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:50:46.266107584 +0000 UTC m=+471.185399371" watchObservedRunningTime="2026-09-12 00:50:46.268081293 +0000 UTC m=+471.187373091"
Sep 12 00:50:47.201890 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:47.201857    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-5z5th"
Sep 12 00:50:57.247587 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:57.247557    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"
Sep 12 00:50:58.489158 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:58.489129    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"]
Sep 12 00:50:58.496796 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:58.496777    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"
Sep 12 00:50:58.504341 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:58.504321    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"]
Sep 12 00:50:58.628406 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:58.628371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f262d78-eb5c-44c6-949a-b1fb3084bc72-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-gr2mk\" (UID: \"1f262d78-eb5c-44c6-949a-b1fb3084bc72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"
Sep 12 00:50:58.628554 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:58.628437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5b9d\" (UniqueName: \"kubernetes.io/projected/1f262d78-eb5c-44c6-949a-b1fb3084bc72-kube-api-access-s5b9d\") pod \"kuadrant-operator-controller-manager-5f685c496f-gr2mk\" (UID: \"1f262d78-eb5c-44c6-949a-b1fb3084bc72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"
Sep 12 00:50:58.728975 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:58.728947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f262d78-eb5c-44c6-949a-b1fb3084bc72-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-gr2mk\" (UID: \"1f262d78-eb5c-44c6-949a-b1fb3084bc72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"
Sep 12 00:50:58.729091 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:58.728989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5b9d\" (UniqueName: \"kubernetes.io/projected/1f262d78-eb5c-44c6-949a-b1fb3084bc72-kube-api-access-s5b9d\") pod \"kuadrant-operator-controller-manager-5f685c496f-gr2mk\" (UID: \"1f262d78-eb5c-44c6-949a-b1fb3084bc72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"
Sep 12 00:50:58.729306 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:58.729286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f262d78-eb5c-44c6-949a-b1fb3084bc72-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-gr2mk\" (UID: \"1f262d78-eb5c-44c6-949a-b1fb3084bc72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"
Sep 12 00:50:58.739866 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:58.739812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5b9d\" (UniqueName: \"kubernetes.io/projected/1f262d78-eb5c-44c6-949a-b1fb3084bc72-kube-api-access-s5b9d\") pod \"kuadrant-operator-controller-manager-5f685c496f-gr2mk\" (UID: \"1f262d78-eb5c-44c6-949a-b1fb3084bc72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"
Sep 12 00:50:58.807720 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:58.807688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"
Sep 12 00:50:59.101142 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.101074    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"]
Sep 12 00:50:59.104569 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:50:59.104539    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f262d78_eb5c_44c6_949a_b1fb3084bc72.slice/crio-e15673f69b2696593fa438da2f54dc0fb80b8ff644f89a2a136ddbe249d12de8 WatchSource:0}: Error finding container e15673f69b2696593fa438da2f54dc0fb80b8ff644f89a2a136ddbe249d12de8: Status 404 returned error can't find the container with id e15673f69b2696593fa438da2f54dc0fb80b8ff644f89a2a136ddbe249d12de8
Sep 12 00:50:59.297067 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.297034    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk" event={"ID":"1f262d78-eb5c-44c6-949a-b1fb3084bc72","Type":"ContainerStarted","Data":"78390bb7099a4b7c40abd32d33c9f696377361cdc75309bcb8076e8c649491a8"}
Sep 12 00:50:59.297243 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.297096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk" event={"ID":"1f262d78-eb5c-44c6-949a-b1fb3084bc72","Type":"ContainerStarted","Data":"e15673f69b2696593fa438da2f54dc0fb80b8ff644f89a2a136ddbe249d12de8"}
Sep 12 00:50:59.297243 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.297189    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"
Sep 12 00:50:59.322751 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.322706    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk" podStartSLOduration=1.322690089 podStartE2EDuration="1.322690089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:50:59.321548864 +0000 UTC m=+484.240840650" watchObservedRunningTime="2026-09-12 00:50:59.322690089 +0000 UTC m=+484.241981876"
Sep 12 00:50:59.451773 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.451737    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"]
Sep 12 00:50:59.460615 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.460571    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"]
Sep 12 00:50:59.472550 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.472519    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"]
Sep 12 00:50:59.472760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.472734    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj" podUID="b01853b3-3cd5-4bdf-9c29-65cffe97c964" containerName="manager" containerID="cri-o://a63c6860e4bef5d8ea728611f876d0d2efbaa8b2b1c56c9f8bd7db4c182b0efd" gracePeriod=2
Sep 12 00:50:59.481274 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.481250    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"]
Sep 12 00:50:59.484827 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:50:59.484788    2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b01853b3-3cd5-4bdf-9c29-65cffe97c964\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-bc4tj\": pods \"kuadrant-operator-controller-manager-76f8f686bf-bc4tj\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"
Sep 12 00:50:59.485221 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.485200    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"]
Sep 12 00:50:59.485662 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.485590    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01853b3-3cd5-4bdf-9c29-65cffe97c964" containerName="manager"
Sep 12 00:50:59.485703 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.485668    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01853b3-3cd5-4bdf-9c29-65cffe97c964" containerName="manager"
Sep 12 00:50:59.485703 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.485691    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f262d78-eb5c-44c6-949a-b1fb3084bc72" containerName="manager"
Sep 12 00:50:59.485703 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.485700    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f262d78-eb5c-44c6-949a-b1fb3084bc72" containerName="manager"
Sep 12 00:50:59.485793 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.485781    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f262d78-eb5c-44c6-949a-b1fb3084bc72" containerName="manager"
Sep 12 00:50:59.485825 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.485792    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b01853b3-3cd5-4bdf-9c29-65cffe97c964" containerName="manager"
Sep 12 00:50:59.503298 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.503273    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"
Sep 12 00:50:59.510239 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.510214    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"]
Sep 12 00:50:59.515751 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.515728    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"]
Sep 12 00:50:59.533395 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.533368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"
Sep 12 00:50:59.543839 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:50:59.543813    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-bc4tj\" is forbidden: User \"system:node:ip-10-0-136-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-121.ec2.internal' and this object" podUID="b01853b3-3cd5-4bdf-9c29-65cffe97c964" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"
Sep 12 00:50:59.545853 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:50:59.545832    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-bc4tj\" is forbidden: User \"system:node:ip-10-0-136-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-121.ec2.internal' and this object" podUID="b01853b3-3cd5-4bdf-9c29-65cffe97c964" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"
Sep 12 00:50:59.549877 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.549854    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"]
Sep 12 00:50:59.637414 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.637316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qfrn\" (UniqueName: \"kubernetes.io/projected/f7e40bbb-f98e-40c9-99c6-bca07c664540-kube-api-access-6qfrn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tssl9\" (UID: \"f7e40bbb-f98e-40c9-99c6-bca07c664540\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"
Sep 12 00:50:59.637414 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.637377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwzhl\" (UniqueName: \"kubernetes.io/projected/8756bdbf-1362-4019-8fad-872deb22317f-kube-api-access-wwzhl\") pod \"kuadrant-operator-controller-manager-5f685c496f-7qpl9\" (UID: \"8756bdbf-1362-4019-8fad-872deb22317f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"
Sep 12 00:50:59.637645 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.637421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7e40bbb-f98e-40c9-99c6-bca07c664540-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tssl9\" (UID: \"f7e40bbb-f98e-40c9-99c6-bca07c664540\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"
Sep 12 00:50:59.637645 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.637462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8756bdbf-1362-4019-8fad-872deb22317f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-7qpl9\" (UID: \"8756bdbf-1362-4019-8fad-872deb22317f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"
Sep 12 00:50:59.738577 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.738553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8756bdbf-1362-4019-8fad-872deb22317f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-7qpl9\" (UID: \"8756bdbf-1362-4019-8fad-872deb22317f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"
Sep 12 00:50:59.738702 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.738637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qfrn\" (UniqueName: \"kubernetes.io/projected/f7e40bbb-f98e-40c9-99c6-bca07c664540-kube-api-access-6qfrn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tssl9\" (UID: \"f7e40bbb-f98e-40c9-99c6-bca07c664540\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"
Sep 12 00:50:59.738702 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.738675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwzhl\" (UniqueName: \"kubernetes.io/projected/8756bdbf-1362-4019-8fad-872deb22317f-kube-api-access-wwzhl\") pod \"kuadrant-operator-controller-manager-5f685c496f-7qpl9\" (UID: \"8756bdbf-1362-4019-8fad-872deb22317f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"
Sep 12 00:50:59.738917 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.738892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7e40bbb-f98e-40c9-99c6-bca07c664540-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tssl9\" (UID: \"f7e40bbb-f98e-40c9-99c6-bca07c664540\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"
Sep 12 00:50:59.739078 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.738962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8756bdbf-1362-4019-8fad-872deb22317f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-7qpl9\" (UID: \"8756bdbf-1362-4019-8fad-872deb22317f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"
Sep 12 00:50:59.739248 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.739227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7e40bbb-f98e-40c9-99c6-bca07c664540-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tssl9\" (UID: \"f7e40bbb-f98e-40c9-99c6-bca07c664540\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"
Sep 12 00:50:59.748116 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.748098    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"
Sep 12 00:50:59.750964 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.750945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwzhl\" (UniqueName: \"kubernetes.io/projected/8756bdbf-1362-4019-8fad-872deb22317f-kube-api-access-wwzhl\") pod \"kuadrant-operator-controller-manager-5f685c496f-7qpl9\" (UID: \"8756bdbf-1362-4019-8fad-872deb22317f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"
Sep 12 00:50:59.751059 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.751032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qfrn\" (UniqueName: \"kubernetes.io/projected/f7e40bbb-f98e-40c9-99c6-bca07c664540-kube-api-access-6qfrn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tssl9\" (UID: \"f7e40bbb-f98e-40c9-99c6-bca07c664540\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"
Sep 12 00:50:59.845838 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.845805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"
Sep 12 00:50:59.852974 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.852955    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"
Sep 12 00:50:59.940945 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.940910    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b01853b3-3cd5-4bdf-9c29-65cffe97c964-extensions-socket-volume\") pod \"b01853b3-3cd5-4bdf-9c29-65cffe97c964\" (UID: \"b01853b3-3cd5-4bdf-9c29-65cffe97c964\") "
Sep 12 00:50:59.941062 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.940995    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjzrw\" (UniqueName: \"kubernetes.io/projected/b01853b3-3cd5-4bdf-9c29-65cffe97c964-kube-api-access-hjzrw\") pod \"b01853b3-3cd5-4bdf-9c29-65cffe97c964\" (UID: \"b01853b3-3cd5-4bdf-9c29-65cffe97c964\") "
Sep 12 00:50:59.941383 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.941303    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01853b3-3cd5-4bdf-9c29-65cffe97c964-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b01853b3-3cd5-4bdf-9c29-65cffe97c964" (UID: "b01853b3-3cd5-4bdf-9c29-65cffe97c964"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:50:59.944486 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:50:59.944445    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01853b3-3cd5-4bdf-9c29-65cffe97c964-kube-api-access-hjzrw" (OuterVolumeSpecName: "kube-api-access-hjzrw") pod "b01853b3-3cd5-4bdf-9c29-65cffe97c964" (UID: "b01853b3-3cd5-4bdf-9c29-65cffe97c964"). InnerVolumeSpecName "kube-api-access-hjzrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:51:00.042241 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:00.042148    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b01853b3-3cd5-4bdf-9c29-65cffe97c964-extensions-socket-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:51:00.042241 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:00.042186    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjzrw\" (UniqueName: \"kubernetes.io/projected/b01853b3-3cd5-4bdf-9c29-65cffe97c964-kube-api-access-hjzrw\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:51:00.151208 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:00.151155    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"]
Sep 12 00:51:00.153137 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:51:00.153100    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8756bdbf_1362_4019_8fad_872deb22317f.slice/crio-cfddc87b911aae88251d2ba697ecb0814ff0afd62bf0c9ee53453ecdec3b4c18 WatchSource:0}: Error finding container cfddc87b911aae88251d2ba697ecb0814ff0afd62bf0c9ee53453ecdec3b4c18: Status 404 returned error can't find the container with id cfddc87b911aae88251d2ba697ecb0814ff0afd62bf0c9ee53453ecdec3b4c18
Sep 12 00:51:00.171780 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:00.171758    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"]
Sep 12 00:51:00.174560 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:51:00.174540    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e40bbb_f98e_40c9_99c6_bca07c664540.slice/crio-b019205c360182691a57dbd77b60fe5aaf6976b1ea2239f61fb6f425a60361aa WatchSource:0}: Error finding container b019205c360182691a57dbd77b60fe5aaf6976b1ea2239f61fb6f425a60361aa: Status 404 returned error can't find the container with id b019205c360182691a57dbd77b60fe5aaf6976b1ea2239f61fb6f425a60361aa
Sep 12 00:51:00.303233 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:00.303158    2579 generic.go:356] "Generic (PLEG): container finished" podID="b01853b3-3cd5-4bdf-9c29-65cffe97c964" containerID="a63c6860e4bef5d8ea728611f876d0d2efbaa8b2b1c56c9f8bd7db4c182b0efd" exitCode=0
Sep 12 00:51:00.303233 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:00.303217    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bc4tj"
Sep 12 00:51:00.303394 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:00.303283    2579 scope.go:117] "RemoveContainer" containerID="a63c6860e4bef5d8ea728611f876d0d2efbaa8b2b1c56c9f8bd7db4c182b0efd"
Sep 12 00:51:00.305387 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:00.305355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9" event={"ID":"8756bdbf-1362-4019-8fad-872deb22317f","Type":"ContainerStarted","Data":"95ec8654a269151c6dd3af735a2891f65960d025e5ee4cd94adceeb23db8f8ee"}
Sep 12 00:51:00.305513 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:00.305413    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9" event={"ID":"8756bdbf-1362-4019-8fad-872deb22317f","Type":"ContainerStarted","Data":"cfddc87b911aae88251d2ba697ecb0814ff0afd62bf0c9ee53453ecdec3b4c18"}
Sep 12 00:51:00.305513 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:00.305463    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"
Sep 12 00:51:00.307537 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:00.307513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9" event={"ID":"f7e40bbb-f98e-40c9-99c6-bca07c664540","Type":"ContainerStarted","Data":"8781dcd0b41fb45ec5ea0ed872f2e13a17028303baad5cf025bb13b8cf9f364f"}
Sep 12 00:51:00.307648 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:00.307557    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9" event={"ID":"f7e40bbb-f98e-40c9-99c6-bca07c664540","Type":"ContainerStarted","Data":"b019205c360182691a57dbd77b60fe5aaf6976b1ea2239f61fb6f425a60361aa"}
Sep 12 00:51:00.307648 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:51:00.307629    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk" container="manager"
Sep 12 00:51:00.319540 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:00.319520    2579 scope.go:117] "RemoveContainer" containerID="a63c6860e4bef5d8ea728611f876d0d2efbaa8b2b1c56c9f8bd7db4c182b0efd"
Sep 12 00:51:00.319870 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:51:00.319852    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63c6860e4bef5d8ea728611f876d0d2efbaa8b2b1c56c9f8bd7db4c182b0efd\": container with ID starting with a63c6860e4bef5d8ea728611f876d0d2efbaa8b2b1c56c9f8bd7db4c182b0efd not found: ID does not exist" containerID="a63c6860e4bef5d8ea728611f876d0d2efbaa8b2b1c56c9f8bd7db4c182b0efd"
Sep 12 00:51:00.319944 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:00.319876    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63c6860e4bef5d8ea728611f876d0d2efbaa8b2b1c56c9f8bd7db4c182b0efd"} err="failed to get container status \"a63c6860e4bef5d8ea728611f876d0d2efbaa8b2b1c56c9f8bd7db4c182b0efd\": rpc error: code = NotFound desc = could not find container \"a63c6860e4bef5d8ea728611f876d0d2efbaa8b2b1c56c9f8bd7db4c182b0efd\": container with ID starting with a63c6860e4bef5d8ea728611f876d0d2efbaa8b2b1c56c9f8bd7db4c182b0efd not found: ID does not exist"
Sep 12 00:51:00.337343 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:00.337307    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9" podStartSLOduration=1.337295493 podStartE2EDuration="1.337295493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:51:00.335556435 +0000 UTC m=+485.254848221" watchObservedRunningTime="2026-09-12 00:51:00.337295493 +0000 UTC m=+485.256587333"
Sep 12 00:51:01.314440 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:01.314399    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk" podUID="1f262d78-eb5c-44c6-949a-b1fb3084bc72" containerName="manager" containerID="cri-o://78390bb7099a4b7c40abd32d33c9f696377361cdc75309bcb8076e8c649491a8" gracePeriod=2
Sep 12 00:51:01.314917 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:01.314723    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"
Sep 12 00:51:01.341486 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:01.341447    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9" podStartSLOduration=2.341428745 podStartE2EDuration="2.341428745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:51:01.339681924 +0000 UTC m=+486.258973711" watchObservedRunningTime="2026-09-12 00:51:01.341428745 +0000 UTC m=+486.260720533"
Sep 12 00:51:01.546500 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:01.546477    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"
Sep 12 00:51:01.551408 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:51:01.551386    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-gr2mk\" is forbidden: User \"system:node:ip-10-0-136-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-121.ec2.internal' and this object" podUID="1f262d78-eb5c-44c6-949a-b1fb3084bc72" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"
Sep 12 00:51:01.554767 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:01.554750    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5b9d\" (UniqueName: \"kubernetes.io/projected/1f262d78-eb5c-44c6-949a-b1fb3084bc72-kube-api-access-s5b9d\") pod \"1f262d78-eb5c-44c6-949a-b1fb3084bc72\" (UID: \"1f262d78-eb5c-44c6-949a-b1fb3084bc72\") "
Sep 12 00:51:01.554823 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:01.554783    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f262d78-eb5c-44c6-949a-b1fb3084bc72-extensions-socket-volume\") pod \"1f262d78-eb5c-44c6-949a-b1fb3084bc72\" (UID: \"1f262d78-eb5c-44c6-949a-b1fb3084bc72\") "
Sep 12 00:51:01.555129 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:01.555110    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f262d78-eb5c-44c6-949a-b1fb3084bc72-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1f262d78-eb5c-44c6-949a-b1fb3084bc72" (UID: "1f262d78-eb5c-44c6-949a-b1fb3084bc72"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:51:01.556626 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:01.556591    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f262d78-eb5c-44c6-949a-b1fb3084bc72-kube-api-access-s5b9d" (OuterVolumeSpecName: "kube-api-access-s5b9d") pod "1f262d78-eb5c-44c6-949a-b1fb3084bc72" (UID: "1f262d78-eb5c-44c6-949a-b1fb3084bc72"). InnerVolumeSpecName "kube-api-access-s5b9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:51:01.589416 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:01.589365    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f262d78-eb5c-44c6-949a-b1fb3084bc72" path="/var/lib/kubelet/pods/1f262d78-eb5c-44c6-949a-b1fb3084bc72/volumes"
Sep 12 00:51:01.589704 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:01.589691    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b01853b3-3cd5-4bdf-9c29-65cffe97c964" path="/var/lib/kubelet/pods/b01853b3-3cd5-4bdf-9c29-65cffe97c964/volumes"
Sep 12 00:51:01.655093 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:01.655070    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f262d78-eb5c-44c6-949a-b1fb3084bc72-extensions-socket-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:51:01.655093 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:01.655090    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5b9d\" (UniqueName: \"kubernetes.io/projected/1f262d78-eb5c-44c6-949a-b1fb3084bc72-kube-api-access-s5b9d\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:51:02.324831 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:02.324797    2579 generic.go:356] "Generic (PLEG): container finished" podID="1f262d78-eb5c-44c6-949a-b1fb3084bc72" containerID="78390bb7099a4b7c40abd32d33c9f696377361cdc75309bcb8076e8c649491a8" exitCode=0
Sep 12 00:51:02.325312 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:02.324849    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-gr2mk"
Sep 12 00:51:02.325312 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:02.324899    2579 scope.go:117] "RemoveContainer" containerID="78390bb7099a4b7c40abd32d33c9f696377361cdc75309bcb8076e8c649491a8"
Sep 12 00:51:02.334015 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:02.333997    2579 scope.go:117] "RemoveContainer" containerID="78390bb7099a4b7c40abd32d33c9f696377361cdc75309bcb8076e8c649491a8"
Sep 12 00:51:02.335112 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:51:02.335086    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78390bb7099a4b7c40abd32d33c9f696377361cdc75309bcb8076e8c649491a8\": container with ID starting with 78390bb7099a4b7c40abd32d33c9f696377361cdc75309bcb8076e8c649491a8 not found: ID does not exist" containerID="78390bb7099a4b7c40abd32d33c9f696377361cdc75309bcb8076e8c649491a8"
Sep 12 00:51:02.335207 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:02.335118    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78390bb7099a4b7c40abd32d33c9f696377361cdc75309bcb8076e8c649491a8"} err="failed to get container status \"78390bb7099a4b7c40abd32d33c9f696377361cdc75309bcb8076e8c649491a8\": rpc error: code = NotFound desc = could not find container \"78390bb7099a4b7c40abd32d33c9f696377361cdc75309bcb8076e8c649491a8\": container with ID starting with 78390bb7099a4b7c40abd32d33c9f696377361cdc75309bcb8076e8c649491a8 not found: ID does not exist"
Sep 12 00:51:11.316633 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.316585    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"
Sep 12 00:51:11.377795 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.377763    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"]
Sep 12 00:51:11.378068 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.377999    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9" podUID="f7e40bbb-f98e-40c9-99c6-bca07c664540" containerName="manager" containerID="cri-o://8781dcd0b41fb45ec5ea0ed872f2e13a17028303baad5cf025bb13b8cf9f364f" gracePeriod=10
Sep 12 00:51:11.379575 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.379553    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"
Sep 12 00:51:11.619291 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.619270    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"
Sep 12 00:51:11.718925 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.718901    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qfrn\" (UniqueName: \"kubernetes.io/projected/f7e40bbb-f98e-40c9-99c6-bca07c664540-kube-api-access-6qfrn\") pod \"f7e40bbb-f98e-40c9-99c6-bca07c664540\" (UID: \"f7e40bbb-f98e-40c9-99c6-bca07c664540\") "
Sep 12 00:51:11.719077 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.718938    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7e40bbb-f98e-40c9-99c6-bca07c664540-extensions-socket-volume\") pod \"f7e40bbb-f98e-40c9-99c6-bca07c664540\" (UID: \"f7e40bbb-f98e-40c9-99c6-bca07c664540\") "
Sep 12 00:51:11.719465 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.719432    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e40bbb-f98e-40c9-99c6-bca07c664540-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f7e40bbb-f98e-40c9-99c6-bca07c664540" (UID: "f7e40bbb-f98e-40c9-99c6-bca07c664540"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:51:11.721174 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.721139    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e40bbb-f98e-40c9-99c6-bca07c664540-kube-api-access-6qfrn" (OuterVolumeSpecName: "kube-api-access-6qfrn") pod "f7e40bbb-f98e-40c9-99c6-bca07c664540" (UID: "f7e40bbb-f98e-40c9-99c6-bca07c664540"). InnerVolumeSpecName "kube-api-access-6qfrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:51:11.721952 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.721913    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"]
Sep 12 00:51:11.722231 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.722214    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e40bbb-f98e-40c9-99c6-bca07c664540" containerName="manager"
Sep 12 00:51:11.722231 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.722231    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e40bbb-f98e-40c9-99c6-bca07c664540" containerName="manager"
Sep 12 00:51:11.722372 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.722291    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e40bbb-f98e-40c9-99c6-bca07c664540" containerName="manager"
Sep 12 00:51:11.725076 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.725058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"
Sep 12 00:51:11.753419 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.753398    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"]
Sep 12 00:51:11.820041 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.820016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4txd7\" (UniqueName: \"kubernetes.io/projected/1a4985c0-2f5f-4392-a8d5-a0eaaf913444-kube-api-access-4txd7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hrv66\" (UID: \"1a4985c0-2f5f-4392-a8d5-a0eaaf913444\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"
Sep 12 00:51:11.820161 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.820063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a4985c0-2f5f-4392-a8d5-a0eaaf913444-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hrv66\" (UID: \"1a4985c0-2f5f-4392-a8d5-a0eaaf913444\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"
Sep 12 00:51:11.820161 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.820131    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qfrn\" (UniqueName: \"kubernetes.io/projected/f7e40bbb-f98e-40c9-99c6-bca07c664540-kube-api-access-6qfrn\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:51:11.820161 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.820142    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7e40bbb-f98e-40c9-99c6-bca07c664540-extensions-socket-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:51:11.920537 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.920513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4txd7\" (UniqueName: \"kubernetes.io/projected/1a4985c0-2f5f-4392-a8d5-a0eaaf913444-kube-api-access-4txd7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hrv66\" (UID: \"1a4985c0-2f5f-4392-a8d5-a0eaaf913444\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"
Sep 12 00:51:11.920646 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.920559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a4985c0-2f5f-4392-a8d5-a0eaaf913444-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hrv66\" (UID: \"1a4985c0-2f5f-4392-a8d5-a0eaaf913444\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"
Sep 12 00:51:11.920892 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.920876    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a4985c0-2f5f-4392-a8d5-a0eaaf913444-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hrv66\" (UID: \"1a4985c0-2f5f-4392-a8d5-a0eaaf913444\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"
Sep 12 00:51:11.940334 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:11.940307    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4txd7\" (UniqueName: \"kubernetes.io/projected/1a4985c0-2f5f-4392-a8d5-a0eaaf913444-kube-api-access-4txd7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hrv66\" (UID: \"1a4985c0-2f5f-4392-a8d5-a0eaaf913444\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"
Sep 12 00:51:12.035889 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:12.035861    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"
Sep 12 00:51:12.308578 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:12.308551    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"]
Sep 12 00:51:12.311214 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:51:12.311189    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4985c0_2f5f_4392_a8d5_a0eaaf913444.slice/crio-255bed2e4e0d3bfa75beeb0c99a5b63cf7355a77e6608545539c6780e3cd520d WatchSource:0}: Error finding container 255bed2e4e0d3bfa75beeb0c99a5b63cf7355a77e6608545539c6780e3cd520d: Status 404 returned error can't find the container with id 255bed2e4e0d3bfa75beeb0c99a5b63cf7355a77e6608545539c6780e3cd520d
Sep 12 00:51:12.369743 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:12.369716    2579 generic.go:356] "Generic (PLEG): container finished" podID="f7e40bbb-f98e-40c9-99c6-bca07c664540" containerID="8781dcd0b41fb45ec5ea0ed872f2e13a17028303baad5cf025bb13b8cf9f364f" exitCode=0
Sep 12 00:51:12.370068 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:12.369766    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"
Sep 12 00:51:12.370068 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:12.369793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9" event={"ID":"f7e40bbb-f98e-40c9-99c6-bca07c664540","Type":"ContainerDied","Data":"8781dcd0b41fb45ec5ea0ed872f2e13a17028303baad5cf025bb13b8cf9f364f"}
Sep 12 00:51:12.370068 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:12.369837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9" event={"ID":"f7e40bbb-f98e-40c9-99c6-bca07c664540","Type":"ContainerDied","Data":"b019205c360182691a57dbd77b60fe5aaf6976b1ea2239f61fb6f425a60361aa"}
Sep 12 00:51:12.370068 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:12.369867    2579 scope.go:117] "RemoveContainer" containerID="8781dcd0b41fb45ec5ea0ed872f2e13a17028303baad5cf025bb13b8cf9f364f"
Sep 12 00:51:12.371743 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:12.371722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66" event={"ID":"1a4985c0-2f5f-4392-a8d5-a0eaaf913444","Type":"ContainerStarted","Data":"32eac081121d4e8724e6365b5402f453490a76341a062f6493fcd4d12c124215"}
Sep 12 00:51:12.371863 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:12.371761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66" event={"ID":"1a4985c0-2f5f-4392-a8d5-a0eaaf913444","Type":"ContainerStarted","Data":"255bed2e4e0d3bfa75beeb0c99a5b63cf7355a77e6608545539c6780e3cd520d"}
Sep 12 00:51:12.371863 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:12.371829    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"
Sep 12 00:51:12.377083 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:12.377064    2579 scope.go:117] "RemoveContainer" containerID="8781dcd0b41fb45ec5ea0ed872f2e13a17028303baad5cf025bb13b8cf9f364f"
Sep 12 00:51:12.377387 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:51:12.377362    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8781dcd0b41fb45ec5ea0ed872f2e13a17028303baad5cf025bb13b8cf9f364f\": container with ID starting with 8781dcd0b41fb45ec5ea0ed872f2e13a17028303baad5cf025bb13b8cf9f364f not found: ID does not exist" containerID="8781dcd0b41fb45ec5ea0ed872f2e13a17028303baad5cf025bb13b8cf9f364f"
Sep 12 00:51:12.377476 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:12.377392    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8781dcd0b41fb45ec5ea0ed872f2e13a17028303baad5cf025bb13b8cf9f364f"} err="failed to get container status \"8781dcd0b41fb45ec5ea0ed872f2e13a17028303baad5cf025bb13b8cf9f364f\": rpc error: code = NotFound desc = could not find container \"8781dcd0b41fb45ec5ea0ed872f2e13a17028303baad5cf025bb13b8cf9f364f\": container with ID starting with 8781dcd0b41fb45ec5ea0ed872f2e13a17028303baad5cf025bb13b8cf9f364f not found: ID does not exist"
Sep 12 00:51:12.411583 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:12.411541    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66" podStartSLOduration=1.411527676 podStartE2EDuration="1.411527676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:51:12.410978909 +0000 UTC m=+497.330270696" watchObservedRunningTime="2026-09-12 00:51:12.411527676 +0000 UTC m=+497.330819462"
Sep 12 00:51:12.434156 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:12.434133    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"]
Sep 12 00:51:12.446842 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:12.446818    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tssl9"]
Sep 12 00:51:13.590989 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:13.590959    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e40bbb-f98e-40c9-99c6-bca07c664540" path="/var/lib/kubelet/pods/f7e40bbb-f98e-40c9-99c6-bca07c664540/volumes"
Sep 12 00:51:23.383035 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:23.383002    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"
Sep 12 00:51:23.444671 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:23.444634    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"]
Sep 12 00:51:23.444952 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:23.444901    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9" podUID="8756bdbf-1362-4019-8fad-872deb22317f" containerName="manager" containerID="cri-o://95ec8654a269151c6dd3af735a2891f65960d025e5ee4cd94adceeb23db8f8ee" gracePeriod=10
Sep 12 00:51:23.683153 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:23.683127    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"
Sep 12 00:51:23.810814 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:23.810787    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwzhl\" (UniqueName: \"kubernetes.io/projected/8756bdbf-1362-4019-8fad-872deb22317f-kube-api-access-wwzhl\") pod \"8756bdbf-1362-4019-8fad-872deb22317f\" (UID: \"8756bdbf-1362-4019-8fad-872deb22317f\") "
Sep 12 00:51:23.810945 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:23.810823    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8756bdbf-1362-4019-8fad-872deb22317f-extensions-socket-volume\") pod \"8756bdbf-1362-4019-8fad-872deb22317f\" (UID: \"8756bdbf-1362-4019-8fad-872deb22317f\") "
Sep 12 00:51:23.811187 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:23.811160    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8756bdbf-1362-4019-8fad-872deb22317f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8756bdbf-1362-4019-8fad-872deb22317f" (UID: "8756bdbf-1362-4019-8fad-872deb22317f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 00:51:23.812718 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:23.812696    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8756bdbf-1362-4019-8fad-872deb22317f-kube-api-access-wwzhl" (OuterVolumeSpecName: "kube-api-access-wwzhl") pod "8756bdbf-1362-4019-8fad-872deb22317f" (UID: "8756bdbf-1362-4019-8fad-872deb22317f"). InnerVolumeSpecName "kube-api-access-wwzhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:51:23.911814 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:23.911782    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwzhl\" (UniqueName: \"kubernetes.io/projected/8756bdbf-1362-4019-8fad-872deb22317f-kube-api-access-wwzhl\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:51:23.911814 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:23.911807    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8756bdbf-1362-4019-8fad-872deb22317f-extensions-socket-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:51:24.430867 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:24.430833    2579 generic.go:356] "Generic (PLEG): container finished" podID="8756bdbf-1362-4019-8fad-872deb22317f" containerID="95ec8654a269151c6dd3af735a2891f65960d025e5ee4cd94adceeb23db8f8ee" exitCode=0
Sep 12 00:51:24.431273 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:24.430873    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"
Sep 12 00:51:24.431273 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:24.430882    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9" event={"ID":"8756bdbf-1362-4019-8fad-872deb22317f","Type":"ContainerDied","Data":"95ec8654a269151c6dd3af735a2891f65960d025e5ee4cd94adceeb23db8f8ee"}
Sep 12 00:51:24.431273 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:24.430931    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9" event={"ID":"8756bdbf-1362-4019-8fad-872deb22317f","Type":"ContainerDied","Data":"cfddc87b911aae88251d2ba697ecb0814ff0afd62bf0c9ee53453ecdec3b4c18"}
Sep 12 00:51:24.431273 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:24.430956    2579 scope.go:117] "RemoveContainer" containerID="95ec8654a269151c6dd3af735a2891f65960d025e5ee4cd94adceeb23db8f8ee"
Sep 12 00:51:24.438929 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:24.438914    2579 scope.go:117] "RemoveContainer" containerID="95ec8654a269151c6dd3af735a2891f65960d025e5ee4cd94adceeb23db8f8ee"
Sep 12 00:51:24.439182 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:51:24.439161    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ec8654a269151c6dd3af735a2891f65960d025e5ee4cd94adceeb23db8f8ee\": container with ID starting with 95ec8654a269151c6dd3af735a2891f65960d025e5ee4cd94adceeb23db8f8ee not found: ID does not exist" containerID="95ec8654a269151c6dd3af735a2891f65960d025e5ee4cd94adceeb23db8f8ee"
Sep 12 00:51:24.439240 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:24.439195    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ec8654a269151c6dd3af735a2891f65960d025e5ee4cd94adceeb23db8f8ee"} err="failed to get container status \"95ec8654a269151c6dd3af735a2891f65960d025e5ee4cd94adceeb23db8f8ee\": rpc error: code = NotFound desc = could not find container \"95ec8654a269151c6dd3af735a2891f65960d025e5ee4cd94adceeb23db8f8ee\": container with ID starting with 95ec8654a269151c6dd3af735a2891f65960d025e5ee4cd94adceeb23db8f8ee not found: ID does not exist"
Sep 12 00:51:24.453411 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:24.453389    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"]
Sep 12 00:51:24.460611 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:24.460582    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-7qpl9"]
Sep 12 00:51:25.589611 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:25.589565    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8756bdbf-1362-4019-8fad-872deb22317f" path="/var/lib/kubelet/pods/8756bdbf-1362-4019-8fad-872deb22317f/volumes"
Sep 12 00:51:27.689209 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.689176    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"]
Sep 12 00:51:27.689665 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.689548    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8756bdbf-1362-4019-8fad-872deb22317f" containerName="manager"
Sep 12 00:51:27.689665 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.689564    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8756bdbf-1362-4019-8fad-872deb22317f" containerName="manager"
Sep 12 00:51:27.689792 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.689697    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8756bdbf-1362-4019-8fad-872deb22317f" containerName="manager"
Sep 12 00:51:27.693966 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.693947    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.696821 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.696801    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9xgcs\""
Sep 12 00:51:27.703539 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.703517    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"]
Sep 12 00:51:27.736718 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.736694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a081c880-b89f-4024-b8f6-e236eaf409f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.736812 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.736723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a081c880-b89f-4024-b8f6-e236eaf409f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.736812 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.736760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a081c880-b89f-4024-b8f6-e236eaf409f7-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.736895 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.736827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-989gb\" (UniqueName: \"kubernetes.io/projected/a081c880-b89f-4024-b8f6-e236eaf409f7-kube-api-access-989gb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.736895 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.736864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a081c880-b89f-4024-b8f6-e236eaf409f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.736968 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.736901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a081c880-b89f-4024-b8f6-e236eaf409f7-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.736968 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.736920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a081c880-b89f-4024-b8f6-e236eaf409f7-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.736968 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.736939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a081c880-b89f-4024-b8f6-e236eaf409f7-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.736968 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.736957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a081c880-b89f-4024-b8f6-e236eaf409f7-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.838019 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.837995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a081c880-b89f-4024-b8f6-e236eaf409f7-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.838100 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.838027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a081c880-b89f-4024-b8f6-e236eaf409f7-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.838100 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.838063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a081c880-b89f-4024-b8f6-e236eaf409f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.838100 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.838094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a081c880-b89f-4024-b8f6-e236eaf409f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.838253 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.838142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a081c880-b89f-4024-b8f6-e236eaf409f7-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.838253 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.838169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-989gb\" (UniqueName: \"kubernetes.io/projected/a081c880-b89f-4024-b8f6-e236eaf409f7-kube-api-access-989gb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.838253 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.838199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a081c880-b89f-4024-b8f6-e236eaf409f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.838394 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.838258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a081c880-b89f-4024-b8f6-e236eaf409f7-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.838394 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.838289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a081c880-b89f-4024-b8f6-e236eaf409f7-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.838487 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.838454    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a081c880-b89f-4024-b8f6-e236eaf409f7-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.838545 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.838521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a081c880-b89f-4024-b8f6-e236eaf409f7-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.838738 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.838714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a081c880-b89f-4024-b8f6-e236eaf409f7-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.838830 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.838759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a081c880-b89f-4024-b8f6-e236eaf409f7-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.838830 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.838815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a081c880-b89f-4024-b8f6-e236eaf409f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.840427 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.840408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a081c880-b89f-4024-b8f6-e236eaf409f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.840540 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.840522    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a081c880-b89f-4024-b8f6-e236eaf409f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.846218 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.846196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a081c880-b89f-4024-b8f6-e236eaf409f7-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:27.846381 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:27.846362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-989gb\" (UniqueName: \"kubernetes.io/projected/a081c880-b89f-4024-b8f6-e236eaf409f7-kube-api-access-989gb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-5hvwt\" (UID: \"a081c880-b89f-4024-b8f6-e236eaf409f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:28.004128 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:28.004062    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:28.278208 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:28.278139    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"]
Sep 12 00:51:28.281116 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:51:28.281083    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda081c880_b89f_4024_b8f6_e236eaf409f7.slice/crio-3bb2a6908a3d0109bcf8c20feb0df442a8b4d7a54e3f7117fa416fa954f7cfb9 WatchSource:0}: Error finding container 3bb2a6908a3d0109bcf8c20feb0df442a8b4d7a54e3f7117fa416fa954f7cfb9: Status 404 returned error can't find the container with id 3bb2a6908a3d0109bcf8c20feb0df442a8b4d7a54e3f7117fa416fa954f7cfb9
Sep 12 00:51:28.283217 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:28.283184    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:51:28.283271 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:28.283248    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:51:28.283309 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:28.283301    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:51:28.455098 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:28.455052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt" event={"ID":"a081c880-b89f-4024-b8f6-e236eaf409f7","Type":"ContainerStarted","Data":"f327840a2912d1d0421e3e694b93f0b4f8d08da3a0614a24014baf522fc2f15e"}
Sep 12 00:51:28.455262 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:28.455122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt" event={"ID":"a081c880-b89f-4024-b8f6-e236eaf409f7","Type":"ContainerStarted","Data":"3bb2a6908a3d0109bcf8c20feb0df442a8b4d7a54e3f7117fa416fa954f7cfb9"}
Sep 12 00:51:28.477064 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:28.477009    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt" podStartSLOduration=1.4769885600000001 podStartE2EDuration="1.47698856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:51:28.474900548 +0000 UTC m=+513.394192336" watchObservedRunningTime="2026-09-12 00:51:28.47698856 +0000 UTC m=+513.396280348"
Sep 12 00:51:29.004256 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:29.004226    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:29.008644 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:29.008623    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:29.459332 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:29.459301    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:51:29.460273 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:51:29.460255    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:52:19.596850 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:19.596818    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f459795f7-k2p55"]
Sep 12 00:52:19.599943 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:19.599925    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 00:52:19.602840 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:19.602818    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 12 00:52:19.602955 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:19.602818    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-484sv\""
Sep 12 00:52:19.604276 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:19.604257    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 12 00:52:19.604372 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:19.604314    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 12 00:52:19.611630 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:19.611592    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f459795f7-k2p55"]
Sep 12 00:52:19.698422 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:19.698393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/215d437d-6ae1-4225-9d31-a6421a20a66a-metrics-certs\") pod \"maas-controller-5f459795f7-k2p55\" (UID: \"215d437d-6ae1-4225-9d31-a6421a20a66a\") " pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 00:52:19.698422 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:19.698427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/215d437d-6ae1-4225-9d31-a6421a20a66a-webhook-cert\") pod \"maas-controller-5f459795f7-k2p55\" (UID: \"215d437d-6ae1-4225-9d31-a6421a20a66a\") " pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 00:52:19.698595 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:19.698448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv7nv\" (UniqueName: \"kubernetes.io/projected/215d437d-6ae1-4225-9d31-a6421a20a66a-kube-api-access-wv7nv\") pod \"maas-controller-5f459795f7-k2p55\" (UID: \"215d437d-6ae1-4225-9d31-a6421a20a66a\") " pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 00:52:19.799551 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:19.799518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv7nv\" (UniqueName: \"kubernetes.io/projected/215d437d-6ae1-4225-9d31-a6421a20a66a-kube-api-access-wv7nv\") pod \"maas-controller-5f459795f7-k2p55\" (UID: \"215d437d-6ae1-4225-9d31-a6421a20a66a\") " pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 00:52:19.799686 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:19.799623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/215d437d-6ae1-4225-9d31-a6421a20a66a-metrics-certs\") pod \"maas-controller-5f459795f7-k2p55\" (UID: \"215d437d-6ae1-4225-9d31-a6421a20a66a\") " pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 00:52:19.799686 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:19.799650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/215d437d-6ae1-4225-9d31-a6421a20a66a-webhook-cert\") pod \"maas-controller-5f459795f7-k2p55\" (UID: \"215d437d-6ae1-4225-9d31-a6421a20a66a\") " pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 00:52:19.799760 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:52:19.799747    2579 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 12 00:52:19.799859 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:52:19.799813    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/215d437d-6ae1-4225-9d31-a6421a20a66a-webhook-cert podName:215d437d-6ae1-4225-9d31-a6421a20a66a nodeName:}" failed. No retries permitted until 2026-09-12 00:52:20.299792544 +0000 UTC m=+565.219084325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/215d437d-6ae1-4225-9d31-a6421a20a66a-webhook-cert") pod "maas-controller-5f459795f7-k2p55" (UID: "215d437d-6ae1-4225-9d31-a6421a20a66a") : secret "maas-controller-webhook-cert" not found
Sep 12 00:52:19.802052 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:19.802034    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/215d437d-6ae1-4225-9d31-a6421a20a66a-metrics-certs\") pod \"maas-controller-5f459795f7-k2p55\" (UID: \"215d437d-6ae1-4225-9d31-a6421a20a66a\") " pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 00:52:19.810009 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:19.809979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv7nv\" (UniqueName: \"kubernetes.io/projected/215d437d-6ae1-4225-9d31-a6421a20a66a-kube-api-access-wv7nv\") pod \"maas-controller-5f459795f7-k2p55\" (UID: \"215d437d-6ae1-4225-9d31-a6421a20a66a\") " pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 00:52:20.304729 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:20.304698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/215d437d-6ae1-4225-9d31-a6421a20a66a-webhook-cert\") pod \"maas-controller-5f459795f7-k2p55\" (UID: \"215d437d-6ae1-4225-9d31-a6421a20a66a\") " pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 00:52:20.306925 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:20.306907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/215d437d-6ae1-4225-9d31-a6421a20a66a-webhook-cert\") pod \"maas-controller-5f459795f7-k2p55\" (UID: \"215d437d-6ae1-4225-9d31-a6421a20a66a\") " pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 00:52:20.509716 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:20.509680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 00:52:20.781071 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:20.781042    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f459795f7-k2p55"]
Sep 12 00:52:20.783892 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:52:20.783865    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215d437d_6ae1_4225_9d31_a6421a20a66a.slice/crio-c854f4c2d59e585f0d51093c4f8e789600204a512a17a59933323d14af6f71fa WatchSource:0}: Error finding container c854f4c2d59e585f0d51093c4f8e789600204a512a17a59933323d14af6f71fa: Status 404 returned error can't find the container with id c854f4c2d59e585f0d51093c4f8e789600204a512a17a59933323d14af6f71fa
Sep 12 00:52:21.669390 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:21.669354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f459795f7-k2p55" event={"ID":"215d437d-6ae1-4225-9d31-a6421a20a66a","Type":"ContainerStarted","Data":"c854f4c2d59e585f0d51093c4f8e789600204a512a17a59933323d14af6f71fa"}
Sep 12 00:52:23.679248 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:23.679216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f459795f7-k2p55" event={"ID":"215d437d-6ae1-4225-9d31-a6421a20a66a","Type":"ContainerStarted","Data":"5a66eeadab7cce9f9fc83e7f27002e7a291b5059cfd1ac82e8335e4fb67cb7ed"}
Sep 12 00:52:23.679660 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:23.679313    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 00:52:23.702135 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:23.702089    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5f459795f7-k2p55" podStartSLOduration=2.488033535 podStartE2EDuration="4.702067814s" totalImagesPullingTime="2.214034279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:52:23.701670726 +0000 UTC m=+568.620962512" watchObservedRunningTime="2026-09-12 00:52:23.702067814 +0000 UTC m=+568.621359601"
Sep 12 00:52:34.686224 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:34.686185    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 00:52:55.524437 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:55.524413    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 00:52:55.524879 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:55.524417    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 00:52:57.907834 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:57.907802    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-plxcj"]
Sep 12 00:52:57.913108 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:57.913084    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-plxcj"
Sep 12 00:52:57.916488 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:57.916467    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 12 00:52:57.916596 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:57.916518    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hvkvx\""
Sep 12 00:52:57.922783 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:57.922761    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-plxcj"]
Sep 12 00:52:57.964948 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:57.964915    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fff3f1-7b21-4c2d-a4fc-33d2a733834e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-plxcj\" (UID: \"f4fff3f1-7b21-4c2d-a4fc-33d2a733834e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-plxcj"
Sep 12 00:52:57.965072 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:57.964961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bssqh\" (UniqueName: \"kubernetes.io/projected/f4fff3f1-7b21-4c2d-a4fc-33d2a733834e-kube-api-access-bssqh\") pod \"payload-processing-5b54b7fbb7-plxcj\" (UID: \"f4fff3f1-7b21-4c2d-a4fc-33d2a733834e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-plxcj"
Sep 12 00:52:58.065767 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:58.065732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fff3f1-7b21-4c2d-a4fc-33d2a733834e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-plxcj\" (UID: \"f4fff3f1-7b21-4c2d-a4fc-33d2a733834e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-plxcj"
Sep 12 00:52:58.065940 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:58.065774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bssqh\" (UniqueName: \"kubernetes.io/projected/f4fff3f1-7b21-4c2d-a4fc-33d2a733834e-kube-api-access-bssqh\") pod \"payload-processing-5b54b7fbb7-plxcj\" (UID: \"f4fff3f1-7b21-4c2d-a4fc-33d2a733834e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-plxcj"
Sep 12 00:52:58.066510 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:58.066484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fff3f1-7b21-4c2d-a4fc-33d2a733834e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-plxcj\" (UID: \"f4fff3f1-7b21-4c2d-a4fc-33d2a733834e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-plxcj"
Sep 12 00:52:58.075880 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:58.075855    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bssqh\" (UniqueName: \"kubernetes.io/projected/f4fff3f1-7b21-4c2d-a4fc-33d2a733834e-kube-api-access-bssqh\") pod \"payload-processing-5b54b7fbb7-plxcj\" (UID: \"f4fff3f1-7b21-4c2d-a4fc-33d2a733834e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-plxcj"
Sep 12 00:52:58.223369 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:58.223328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-plxcj"
Sep 12 00:52:58.495562 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:58.495483    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-plxcj"]
Sep 12 00:52:58.498553 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:52:58.498517    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fff3f1_7b21_4c2d_a4fc_33d2a733834e.slice/crio-b4eeae08016a066fdc3cc4f9a1842d3936a335ff92c84301d04531d6f492fc36 WatchSource:0}: Error finding container b4eeae08016a066fdc3cc4f9a1842d3936a335ff92c84301d04531d6f492fc36: Status 404 returned error can't find the container with id b4eeae08016a066fdc3cc4f9a1842d3936a335ff92c84301d04531d6f492fc36
Sep 12 00:52:58.819371 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:52:58.819264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-plxcj" event={"ID":"f4fff3f1-7b21-4c2d-a4fc-33d2a733834e","Type":"ContainerStarted","Data":"b4eeae08016a066fdc3cc4f9a1842d3936a335ff92c84301d04531d6f492fc36"}
Sep 12 00:53:00.831687 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:00.831648    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-plxcj" event={"ID":"f4fff3f1-7b21-4c2d-a4fc-33d2a733834e","Type":"ContainerStarted","Data":"a0ea1f054f1fda56d3da9228bf8c4750901359ff2e6a393cd2f77614298c701b"}
Sep 12 00:53:00.832109 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:00.831716    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-plxcj"
Sep 12 00:53:00.850391 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:00.850345    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-plxcj" podStartSLOduration=2.271574595 podStartE2EDuration="3.850330557s" totalImagesPullingTime="1.578755962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:53:00.84905645 +0000 UTC m=+605.768348240" watchObservedRunningTime="2026-09-12 00:53:00.850330557 +0000 UTC m=+605.769622344"
Sep 12 00:53:07.862542 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:07.862510    2579 generic.go:356] "Generic (PLEG): container finished" podID="a081c880-b89f-4024-b8f6-e236eaf409f7" containerID="f327840a2912d1d0421e3e694b93f0b4f8d08da3a0614a24014baf522fc2f15e" exitCode=0
Sep 12 00:53:07.862943 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:07.862582    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt" event={"ID":"a081c880-b89f-4024-b8f6-e236eaf409f7","Type":"ContainerDied","Data":"f327840a2912d1d0421e3e694b93f0b4f8d08da3a0614a24014baf522fc2f15e"}
Sep 12 00:53:07.862943 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:07.862932    2579 scope.go:117] "RemoveContainer" containerID="f327840a2912d1d0421e3e694b93f0b4f8d08da3a0614a24014baf522fc2f15e"
Sep 12 00:53:07.863947 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:07.863924    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:53:07.864005 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:07.863987    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:53:07.864039 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:07.864017    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:53:08.868527 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:08.868494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt" event={"ID":"a081c880-b89f-4024-b8f6-e236eaf409f7","Type":"ContainerStarted","Data":"7cc102c6b91e8edfbcd9017fd8b0cb73cd596ef9f63839870d8eaec9f18660eb"}
Sep 12 00:53:09.004244 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:09.004209    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:53:09.004438 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:09.004387    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-5hvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Sep 12 00:53:09.004509 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:09.004462    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt" podUID="a081c880-b89f-4024-b8f6-e236eaf409f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Sep 12 00:53:10.005650 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:10.005191    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-5hvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Sep 12 00:53:10.005650 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:10.005262    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt" podUID="a081c880-b89f-4024-b8f6-e236eaf409f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Sep 12 00:53:11.005315 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:11.005275    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-5hvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Sep 12 00:53:11.005520 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:11.005350    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt" podUID="a081c880-b89f-4024-b8f6-e236eaf409f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Sep 12 00:53:11.838236 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:11.838203    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-plxcj"
Sep 12 00:53:12.005025 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:12.004986    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-5hvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Sep 12 00:53:12.005330 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:12.005292    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt" podUID="a081c880-b89f-4024-b8f6-e236eaf409f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Sep 12 00:53:13.005334 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:13.004913    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-5hvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Sep 12 00:53:13.005334 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:13.004973    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt" podUID="a081c880-b89f-4024-b8f6-e236eaf409f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Sep 12 00:53:14.008471 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:14.008442    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:53:14.008852 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:14.008796    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:53:14.009503 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:14.009481    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-5hvwt"
Sep 12 00:53:20.638138 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.638064    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"]
Sep 12 00:53:20.641251 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.641229    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.643886 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.643865    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 12 00:53:20.643992 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.643965    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 12 00:53:20.644787 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.644765    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6nb8d\""
Sep 12 00:53:20.644880 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.644804    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 12 00:53:20.651933 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.651913    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"]
Sep 12 00:53:20.736075 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.736044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/770731cd-3368-43f7-8348-da4d3786353c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.736210 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.736091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/770731cd-3368-43f7-8348-da4d3786353c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.736210 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.736149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/770731cd-3368-43f7-8348-da4d3786353c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.736210 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.736175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/770731cd-3368-43f7-8348-da4d3786353c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.736349 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.736240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/770731cd-3368-43f7-8348-da4d3786353c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.736349 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.736268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkf97\" (UniqueName: \"kubernetes.io/projected/770731cd-3368-43f7-8348-da4d3786353c-kube-api-access-kkf97\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.836639 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.836587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/770731cd-3368-43f7-8348-da4d3786353c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.836639 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.836636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/770731cd-3368-43f7-8348-da4d3786353c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.836857 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.836661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/770731cd-3368-43f7-8348-da4d3786353c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.836857 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.836694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkf97\" (UniqueName: \"kubernetes.io/projected/770731cd-3368-43f7-8348-da4d3786353c-kube-api-access-kkf97\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.836857 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.836754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/770731cd-3368-43f7-8348-da4d3786353c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.836857 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.836775    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/770731cd-3368-43f7-8348-da4d3786353c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.837171 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.837145    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/770731cd-3368-43f7-8348-da4d3786353c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.837292 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.837215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/770731cd-3368-43f7-8348-da4d3786353c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.837292 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.837264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/770731cd-3368-43f7-8348-da4d3786353c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.838920 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.838902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/770731cd-3368-43f7-8348-da4d3786353c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.839396 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.839380    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/770731cd-3368-43f7-8348-da4d3786353c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.844293 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.844271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkf97\" (UniqueName: \"kubernetes.io/projected/770731cd-3368-43f7-8348-da4d3786353c-kube-api-access-kkf97\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj\" (UID: \"770731cd-3368-43f7-8348-da4d3786353c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:20.951822 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:20.951796    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:21.226671 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:21.226646    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"]
Sep 12 00:53:21.228715 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:53:21.228687    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770731cd_3368_43f7_8348_da4d3786353c.slice/crio-7db975b60f70c56a284a9d73ae74c4ea522c798f064fd179e788058c7955ce0a WatchSource:0}: Error finding container 7db975b60f70c56a284a9d73ae74c4ea522c798f064fd179e788058c7955ce0a: Status 404 returned error can't find the container with id 7db975b60f70c56a284a9d73ae74c4ea522c798f064fd179e788058c7955ce0a
Sep 12 00:53:21.229919 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:21.229900    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:53:21.926244 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:21.926208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj" event={"ID":"770731cd-3368-43f7-8348-da4d3786353c","Type":"ContainerStarted","Data":"7db975b60f70c56a284a9d73ae74c4ea522c798f064fd179e788058c7955ce0a"}
Sep 12 00:53:22.932838 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:22.932805    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj" event={"ID":"770731cd-3368-43f7-8348-da4d3786353c","Type":"ContainerStarted","Data":"8b6d823a5ac66e7951a82294c5845a7fd5b87c6ab63dc85757e7d0c5f8ce0a3e"}
Sep 12 00:53:22.951337 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:22.951287    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj" podStartSLOduration=1.7705847399999999 podStartE2EDuration="2.951271254s" totalImagesPullingTime="1.180686514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:53:22.950587557 +0000 UTC m=+627.869879343" watchObservedRunningTime="2026-09-12 00:53:22.951271254 +0000 UTC m=+627.870563088"
Sep 12 00:53:30.952212 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:30.952179    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:30.952642 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:30.952236    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:30.964299 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:30.964278    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:53:30.980835 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:53:30.980810    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj"
Sep 12 00:54:12.827618 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.827579    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"]
Sep 12 00:54:12.831710 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.831690    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.833991 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.833969    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 12 00:54:12.835158 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.835140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e81e26bb-c791-4118-8366-02743ce76d5e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.835227 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.835168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e81e26bb-c791-4118-8366-02743ce76d5e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.835227 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.835193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhsdv\" (UniqueName: \"kubernetes.io/projected/e81e26bb-c791-4118-8366-02743ce76d5e-kube-api-access-lhsdv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.835227 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.835221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e81e26bb-c791-4118-8366-02743ce76d5e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.835346 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.835272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e81e26bb-c791-4118-8366-02743ce76d5e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.835346 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.835301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e81e26bb-c791-4118-8366-02743ce76d5e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.841979 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.841958    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"]
Sep 12 00:54:12.936310 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.936267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e81e26bb-c791-4118-8366-02743ce76d5e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.936310 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.936309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e81e26bb-c791-4118-8366-02743ce76d5e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.936521 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.936351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhsdv\" (UniqueName: \"kubernetes.io/projected/e81e26bb-c791-4118-8366-02743ce76d5e-kube-api-access-lhsdv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.936521 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.936387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e81e26bb-c791-4118-8366-02743ce76d5e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.936521 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.936422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e81e26bb-c791-4118-8366-02743ce76d5e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.936521 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.936463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e81e26bb-c791-4118-8366-02743ce76d5e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.936852 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.936822    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e81e26bb-c791-4118-8366-02743ce76d5e-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.936975 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.936848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e81e26bb-c791-4118-8366-02743ce76d5e-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.936975 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.936892    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e81e26bb-c791-4118-8366-02743ce76d5e-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.938843 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.938820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e81e26bb-c791-4118-8366-02743ce76d5e-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.938988 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.938866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e81e26bb-c791-4118-8366-02743ce76d5e-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:12.946639 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:12.946597    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhsdv\" (UniqueName: \"kubernetes.io/projected/e81e26bb-c791-4118-8366-02743ce76d5e-kube-api-access-lhsdv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9\" (UID: \"e81e26bb-c791-4118-8366-02743ce76d5e\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:13.142196 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:13.142113    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:13.412438 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:13.412409    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"]
Sep 12 00:54:13.415170 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:54:13.415131    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81e26bb_c791_4118_8366_02743ce76d5e.slice/crio-812c3ebd171acfb805acad8e043601fd67388bf8267613e9e9a486e636dbfe16 WatchSource:0}: Error finding container 812c3ebd171acfb805acad8e043601fd67388bf8267613e9e9a486e636dbfe16: Status 404 returned error can't find the container with id 812c3ebd171acfb805acad8e043601fd67388bf8267613e9e9a486e636dbfe16
Sep 12 00:54:14.147526 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:14.147483    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9" event={"ID":"e81e26bb-c791-4118-8366-02743ce76d5e","Type":"ContainerStarted","Data":"e4763ca7b8f2ee9db28353e65b022169a7436830f07de81e96ec1c71ac227f56"}
Sep 12 00:54:14.147987 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:14.147552    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9" event={"ID":"e81e26bb-c791-4118-8366-02743ce76d5e","Type":"ContainerStarted","Data":"812c3ebd171acfb805acad8e043601fd67388bf8267613e9e9a486e636dbfe16"}
Sep 12 00:54:14.166617 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:14.166556    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9" podStartSLOduration=1.976254274 podStartE2EDuration="2.166541198s" totalImagesPullingTime="190.286924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:54:14.165379248 +0000 UTC m=+679.084671038" watchObservedRunningTime="2026-09-12 00:54:14.166541198 +0000 UTC m=+679.085832985"
Sep 12 00:54:23.142238 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:23.142203    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:23.142703 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:23.142260    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:23.154668 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:23.154644    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:54:23.194562 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:54:23.194537    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9"
Sep 12 00:56:06.379295 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.379261    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"]
Sep 12 00:56:06.382666 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.382644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.385289 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.385266    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5cee89-openshift-default-dockercfg-qnxmd\""
Sep 12 00:56:06.395457 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.395436    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"]
Sep 12 00:56:06.506504 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.506469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-envoy\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.506504 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.506501    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-data\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.506765 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.506522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-workload-socket\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.506765 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.506538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istiod-ca-cert\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.506765 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.506559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-token\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.506765 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.506632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-workload-certs\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.506765 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.506672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-podinfo\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.506765 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.506698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-credential-socket\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.506765 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.506734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh7wk\" (UniqueName: \"kubernetes.io/projected/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-kube-api-access-rh7wk\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.607346 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.607300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-credential-socket\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.607521 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.607369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh7wk\" (UniqueName: \"kubernetes.io/projected/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-kube-api-access-rh7wk\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.607521 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.607469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-envoy\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.607521 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.607495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-data\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.607723 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.607522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-workload-socket\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.607723 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.607547    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istiod-ca-cert\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.607723 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.607572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-token\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.607723 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.607617    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-workload-certs\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.607723 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.607651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-podinfo\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.607723 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.607719    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-credential-socket\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.608032 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.608014    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-workload-socket\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.608391 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.608369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-data\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.608545 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.608521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istiod-ca-cert\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.608632 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.608457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-workload-certs\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.610553 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.610508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-envoy\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.610988 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.610814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-podinfo\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.624442 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.624423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-token\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.644969 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.644915    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"]
Sep 12 00:56:06.648128 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.648105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh7wk\" (UniqueName: \"kubernetes.io/projected/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-kube-api-access-rh7wk\") pod \"e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") " pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.649065 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.649051    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.653895 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.653870    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f19fd1-openshift-default-dockercfg-lbjmq\""
Sep 12 00:56:06.670818 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.670796    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"]
Sep 12 00:56:06.695511 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.694021    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:06.708810 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.708781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-istio-envoy\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.708910 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.708829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171640c8-513b-4b58-a865-08d1cf2faf75-istio-podinfo\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.708910 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.708884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-workload-socket\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.708910 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.708906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-credential-socket\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.709152 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.708935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-istio-data\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.709152 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.708969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171640c8-513b-4b58-a865-08d1cf2faf75-istiod-ca-cert\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.709152 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.708993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8v59\" (UniqueName: \"kubernetes.io/projected/171640c8-513b-4b58-a865-08d1cf2faf75-kube-api-access-g8v59\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.709152 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.709021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171640c8-513b-4b58-a865-08d1cf2faf75-istio-token\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.709152 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.709048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-workload-certs\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.810789 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.810757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-workload-certs\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.810979 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.810869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-istio-envoy\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.810979 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.810906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171640c8-513b-4b58-a865-08d1cf2faf75-istio-podinfo\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.810979 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.810941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-workload-socket\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.810979 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.810965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-credential-socket\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.811190 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.811001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-istio-data\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.811190 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.811043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171640c8-513b-4b58-a865-08d1cf2faf75-istiod-ca-cert\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.811190 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.811068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8v59\" (UniqueName: \"kubernetes.io/projected/171640c8-513b-4b58-a865-08d1cf2faf75-kube-api-access-g8v59\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.811190 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.811098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171640c8-513b-4b58-a865-08d1cf2faf75-istio-token\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.811390 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.811191    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-workload-certs\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.811449 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.811432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-istio-data\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.811866 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.811696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-workload-socket\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.811866 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.811811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-credential-socket\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.812052 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.811936    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171640c8-513b-4b58-a865-08d1cf2faf75-istiod-ca-cert\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.813593 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.813573    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171640c8-513b-4b58-a865-08d1cf2faf75-istio-podinfo\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.813593 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.813593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-istio-envoy\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.824474 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.824451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171640c8-513b-4b58-a865-08d1cf2faf75-istio-token\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.824585 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.824461    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8v59\" (UniqueName: \"kubernetes.io/projected/171640c8-513b-4b58-a865-08d1cf2faf75-kube-api-access-g8v59\") pod \"e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") " pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.963187 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.963161    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:06.973169 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.973147    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"]
Sep 12 00:56:06.975010 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:56:06.974985    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd61045_fc5a_46f4_adf5_50d41b74a5a6.slice/crio-80e35f443ecd978b1bfde7432f24db6b5e695455a92fbd108177eac01f85424d WatchSource:0}: Error finding container 80e35f443ecd978b1bfde7432f24db6b5e695455a92fbd108177eac01f85424d: Status 404 returned error can't find the container with id 80e35f443ecd978b1bfde7432f24db6b5e695455a92fbd108177eac01f85424d
Sep 12 00:56:06.977303 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.977273    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:56:06.977398 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.977337    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:56:06.977398 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:06.977382    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:56:07.240526 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:07.240446    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"]
Sep 12 00:56:07.243629 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:56:07.243583    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171640c8_513b_4b58_a865_08d1cf2faf75.slice/crio-a083122083854642b6e2b3c74a8a71c02ae06d4edb2e8067c94dd67a5127cf06 WatchSource:0}: Error finding container a083122083854642b6e2b3c74a8a71c02ae06d4edb2e8067c94dd67a5127cf06: Status 404 returned error can't find the container with id a083122083854642b6e2b3c74a8a71c02ae06d4edb2e8067c94dd67a5127cf06
Sep 12 00:56:07.246286 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:07.246255    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:56:07.246411 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:07.246333    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:56:07.246411 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:07.246378    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:56:07.618836 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:07.618756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc" event={"ID":"171640c8-513b-4b58-a865-08d1cf2faf75","Type":"ContainerStarted","Data":"771af17de8c7920efbfb1df998f77e8819bc1a4618296cd30333f3b9d971f685"}
Sep 12 00:56:07.618836 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:07.618809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc" event={"ID":"171640c8-513b-4b58-a865-08d1cf2faf75","Type":"ContainerStarted","Data":"a083122083854642b6e2b3c74a8a71c02ae06d4edb2e8067c94dd67a5127cf06"}
Sep 12 00:56:07.620536 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:07.620509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7" event={"ID":"7dd61045-fc5a-46f4-adf5-50d41b74a5a6","Type":"ContainerStarted","Data":"5bfb7cc88ad20236757040a0d179722cf3f6c65d0ef544c635be3dc7b64c8d18"}
Sep 12 00:56:07.620657 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:07.620563    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7" event={"ID":"7dd61045-fc5a-46f4-adf5-50d41b74a5a6","Type":"ContainerStarted","Data":"80e35f443ecd978b1bfde7432f24db6b5e695455a92fbd108177eac01f85424d"}
Sep 12 00:56:07.644422 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:07.644373    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc" podStartSLOduration=1.6443522160000001 podStartE2EDuration="1.644352216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:07.642950514 +0000 UTC m=+792.562242302" watchObservedRunningTime="2026-09-12 00:56:07.644352216 +0000 UTC m=+792.563644024"
Sep 12 00:56:07.668934 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:07.668893    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7" podStartSLOduration=1.668879774 podStartE2EDuration="1.668879774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:07.667006957 +0000 UTC m=+792.586298743" watchObservedRunningTime="2026-09-12 00:56:07.668879774 +0000 UTC m=+792.588171572"
Sep 12 00:56:07.696512 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:07.696490    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:07.963733 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:07.963694    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:08.701060 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:08.701031    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:08.969291 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:08.969212    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:09.629943 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:09.629914    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:09.630205 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:09.630179    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:09.630701 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:09.630681    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:09.630871 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:09.630854    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:11.972833 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:11.972794    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"]
Sep 12 00:56:11.976567 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:11.976545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"
Sep 12 00:56:11.981251 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:11.981235    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-f19fd1\""
Sep 12 00:56:12.002379 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.002353    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"]
Sep 12 00:56:12.004055 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.004032    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"]
Sep 12 00:56:12.007272 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.007256    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"
Sep 12 00:56:12.010380 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.010350    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-f19fd1-dockercfg-9jt4x\""
Sep 12 00:56:12.018799 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.018777    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"]
Sep 12 00:56:12.059485 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.059461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n2cj\" (UniqueName: \"kubernetes.io/projected/d98b6109-80a7-4ad8-8795-47430bea0a98-kube-api-access-8n2cj\") pod \"payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p\" (UID: \"d98b6109-80a7-4ad8-8795-47430bea0a98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"
Sep 12 00:56:12.059635 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.059493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98b6109-80a7-4ad8-8795-47430bea0a98-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p\" (UID: \"d98b6109-80a7-4ad8-8795-47430bea0a98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"
Sep 12 00:56:12.160303 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.160258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n2cj\" (UniqueName: \"kubernetes.io/projected/d98b6109-80a7-4ad8-8795-47430bea0a98-kube-api-access-8n2cj\") pod \"payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p\" (UID: \"d98b6109-80a7-4ad8-8795-47430bea0a98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"
Sep 12 00:56:12.160481 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.160316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98b6109-80a7-4ad8-8795-47430bea0a98-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p\" (UID: \"d98b6109-80a7-4ad8-8795-47430bea0a98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"
Sep 12 00:56:12.160481 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.160351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrjx2\" (UniqueName: \"kubernetes.io/projected/4d0c2aa1-7e24-4d5b-a711-1f8657dbe378-kube-api-access-vrjx2\") pod \"payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb\" (UID: \"4d0c2aa1-7e24-4d5b-a711-1f8657dbe378\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"
Sep 12 00:56:12.160481 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.160436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0c2aa1-7e24-4d5b-a711-1f8657dbe378-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb\" (UID: \"4d0c2aa1-7e24-4d5b-a711-1f8657dbe378\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"
Sep 12 00:56:12.161410 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.161380    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98b6109-80a7-4ad8-8795-47430bea0a98-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p\" (UID: \"d98b6109-80a7-4ad8-8795-47430bea0a98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"
Sep 12 00:56:12.179594 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.179563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n2cj\" (UniqueName: \"kubernetes.io/projected/d98b6109-80a7-4ad8-8795-47430bea0a98-kube-api-access-8n2cj\") pod \"payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p\" (UID: \"d98b6109-80a7-4ad8-8795-47430bea0a98\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"
Sep 12 00:56:12.261204 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.261128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0c2aa1-7e24-4d5b-a711-1f8657dbe378-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb\" (UID: \"4d0c2aa1-7e24-4d5b-a711-1f8657dbe378\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"
Sep 12 00:56:12.261343 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.261227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrjx2\" (UniqueName: \"kubernetes.io/projected/4d0c2aa1-7e24-4d5b-a711-1f8657dbe378-kube-api-access-vrjx2\") pod \"payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb\" (UID: \"4d0c2aa1-7e24-4d5b-a711-1f8657dbe378\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"
Sep 12 00:56:12.261904 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.261885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0c2aa1-7e24-4d5b-a711-1f8657dbe378-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb\" (UID: \"4d0c2aa1-7e24-4d5b-a711-1f8657dbe378\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"
Sep 12 00:56:12.287025 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.287000    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"
Sep 12 00:56:12.290914 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.290886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrjx2\" (UniqueName: \"kubernetes.io/projected/4d0c2aa1-7e24-4d5b-a711-1f8657dbe378-kube-api-access-vrjx2\") pod \"payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb\" (UID: \"4d0c2aa1-7e24-4d5b-a711-1f8657dbe378\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"
Sep 12 00:56:12.318361 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.318336    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"
Sep 12 00:56:12.575718 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.575694    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"]
Sep 12 00:56:12.577722 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:56:12.577699    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98b6109_80a7_4ad8_8795_47430bea0a98.slice/crio-97b67ac3e60098e93daeef253e0296763b86b131357e1399c76e6b4a6c768677 WatchSource:0}: Error finding container 97b67ac3e60098e93daeef253e0296763b86b131357e1399c76e6b4a6c768677: Status 404 returned error can't find the container with id 97b67ac3e60098e93daeef253e0296763b86b131357e1399c76e6b4a6c768677
Sep 12 00:56:12.605663 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.605596    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"]
Sep 12 00:56:12.609637 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:56:12.609591    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0c2aa1_7e24_4d5b_a711_1f8657dbe378.slice/crio-8259ca8015f9a06205a4f8451e9091ad008ffa2fa0aabc7ed2b0e06d0c8273e6 WatchSource:0}: Error finding container 8259ca8015f9a06205a4f8451e9091ad008ffa2fa0aabc7ed2b0e06d0c8273e6: Status 404 returned error can't find the container with id 8259ca8015f9a06205a4f8451e9091ad008ffa2fa0aabc7ed2b0e06d0c8273e6
Sep 12 00:56:12.646897 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.646873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb" event={"ID":"4d0c2aa1-7e24-4d5b-a711-1f8657dbe378","Type":"ContainerStarted","Data":"8259ca8015f9a06205a4f8451e9091ad008ffa2fa0aabc7ed2b0e06d0c8273e6"}
Sep 12 00:56:12.648505 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:12.648486    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p" event={"ID":"d98b6109-80a7-4ad8-8795-47430bea0a98","Type":"ContainerStarted","Data":"97b67ac3e60098e93daeef253e0296763b86b131357e1399c76e6b4a6c768677"}
Sep 12 00:56:13.241704 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.241671    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"]
Sep 12 00:56:13.250695 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.250669    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"
Sep 12 00:56:13.254057 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.254035    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5cee89\""
Sep 12 00:56:13.259824 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.259788    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"]
Sep 12 00:56:13.375018 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.374980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc7kr\" (UniqueName: \"kubernetes.io/projected/908baaef-2748-459c-bd57-1f0d994c29bd-kube-api-access-fc7kr\") pod \"payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp\" (UID: \"908baaef-2748-459c-bd57-1f0d994c29bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"
Sep 12 00:56:13.375189 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.375036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/908baaef-2748-459c-bd57-1f0d994c29bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp\" (UID: \"908baaef-2748-459c-bd57-1f0d994c29bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"
Sep 12 00:56:13.475791 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.475754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc7kr\" (UniqueName: \"kubernetes.io/projected/908baaef-2748-459c-bd57-1f0d994c29bd-kube-api-access-fc7kr\") pod \"payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp\" (UID: \"908baaef-2748-459c-bd57-1f0d994c29bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"
Sep 12 00:56:13.475791 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.475793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/908baaef-2748-459c-bd57-1f0d994c29bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp\" (UID: \"908baaef-2748-459c-bd57-1f0d994c29bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"
Sep 12 00:56:13.476447 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.476429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/908baaef-2748-459c-bd57-1f0d994c29bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp\" (UID: \"908baaef-2748-459c-bd57-1f0d994c29bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"
Sep 12 00:56:13.484795 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.484770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc7kr\" (UniqueName: \"kubernetes.io/projected/908baaef-2748-459c-bd57-1f0d994c29bd-kube-api-access-fc7kr\") pod \"payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp\" (UID: \"908baaef-2748-459c-bd57-1f0d994c29bd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"
Sep 12 00:56:13.562848 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.562773    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"
Sep 12 00:56:13.657222 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.656912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb" event={"ID":"4d0c2aa1-7e24-4d5b-a711-1f8657dbe378","Type":"ContainerStarted","Data":"cdd46b6559a52acb821c4202c9146e6b781c9583d00feff73f8a99641906d544"}
Sep 12 00:56:13.657222 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.656984    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"
Sep 12 00:56:13.658961 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.658932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p" event={"ID":"d98b6109-80a7-4ad8-8795-47430bea0a98","Type":"ContainerStarted","Data":"f6e860014bf65138a45b53978bab43efbcb98a105792281c42ea114f5b67f94a"}
Sep 12 00:56:13.659095 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.658987    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"
Sep 12 00:56:13.687074 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.687031    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb" podStartSLOduration=2.687016394 podStartE2EDuration="2.687016394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:13.685642655 +0000 UTC m=+798.604934435" watchObservedRunningTime="2026-09-12 00:56:13.687016394 +0000 UTC m=+798.606308181"
Sep 12 00:56:13.705946 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.705875    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p" podStartSLOduration=2.705856152 podStartE2EDuration="2.705856152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:13.705147542 +0000 UTC m=+798.624439355" watchObservedRunningTime="2026-09-12 00:56:13.705856152 +0000 UTC m=+798.625147940"
Sep 12 00:56:13.851276 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:13.851199    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"]
Sep 12 00:56:13.854628 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:56:13.854586    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908baaef_2748_459c_bd57_1f0d994c29bd.slice/crio-9113c2a4466b7051d1e8379c0fa9641fe0b7ce21265c275b30a4473574610bf8 WatchSource:0}: Error finding container 9113c2a4466b7051d1e8379c0fa9641fe0b7ce21265c275b30a4473574610bf8: Status 404 returned error can't find the container with id 9113c2a4466b7051d1e8379c0fa9641fe0b7ce21265c275b30a4473574610bf8
Sep 12 00:56:14.329796 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.329767    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"]
Sep 12 00:56:14.333006 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.332990    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"
Sep 12 00:56:14.335686 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.335670    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-10ba6f\""
Sep 12 00:56:14.335742 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.335723    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-10ba6f-dockercfg-kdnzb\""
Sep 12 00:56:14.344549 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.344527    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"]
Sep 12 00:56:14.386787 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.386758    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"]
Sep 12 00:56:14.389920 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.389906    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"
Sep 12 00:56:14.400328 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.400308    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"]
Sep 12 00:56:14.485278 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.485246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdca3a0b-aea6-42f7-8550-4a0f4d23209b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs\" (UID: \"bdca3a0b-aea6-42f7-8550-4a0f4d23209b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"
Sep 12 00:56:14.485453 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.485295    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968de32b-1dea-422a-a1aa-dc4f78412fbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl\" (UID: \"968de32b-1dea-422a-a1aa-dc4f78412fbb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"
Sep 12 00:56:14.485453 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.485316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnr6r\" (UniqueName: \"kubernetes.io/projected/968de32b-1dea-422a-a1aa-dc4f78412fbb-kube-api-access-dnr6r\") pod \"payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl\" (UID: \"968de32b-1dea-422a-a1aa-dc4f78412fbb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"
Sep 12 00:56:14.485651 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.485442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r7q8\" (UniqueName: \"kubernetes.io/projected/bdca3a0b-aea6-42f7-8550-4a0f4d23209b-kube-api-access-2r7q8\") pod \"payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs\" (UID: \"bdca3a0b-aea6-42f7-8550-4a0f4d23209b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"
Sep 12 00:56:14.586612 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.586513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdca3a0b-aea6-42f7-8550-4a0f4d23209b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs\" (UID: \"bdca3a0b-aea6-42f7-8550-4a0f4d23209b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"
Sep 12 00:56:14.586612 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.586564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968de32b-1dea-422a-a1aa-dc4f78412fbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl\" (UID: \"968de32b-1dea-422a-a1aa-dc4f78412fbb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"
Sep 12 00:56:14.586612 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.586581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnr6r\" (UniqueName: \"kubernetes.io/projected/968de32b-1dea-422a-a1aa-dc4f78412fbb-kube-api-access-dnr6r\") pod \"payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl\" (UID: \"968de32b-1dea-422a-a1aa-dc4f78412fbb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"
Sep 12 00:56:14.586880 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.586647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r7q8\" (UniqueName: \"kubernetes.io/projected/bdca3a0b-aea6-42f7-8550-4a0f4d23209b-kube-api-access-2r7q8\") pod \"payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs\" (UID: \"bdca3a0b-aea6-42f7-8550-4a0f4d23209b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"
Sep 12 00:56:14.587271 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.587248    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968de32b-1dea-422a-a1aa-dc4f78412fbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl\" (UID: \"968de32b-1dea-422a-a1aa-dc4f78412fbb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"
Sep 12 00:56:14.587351 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.587257    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdca3a0b-aea6-42f7-8550-4a0f4d23209b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs\" (UID: \"bdca3a0b-aea6-42f7-8550-4a0f4d23209b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"
Sep 12 00:56:14.595178 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.595156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnr6r\" (UniqueName: \"kubernetes.io/projected/968de32b-1dea-422a-a1aa-dc4f78412fbb-kube-api-access-dnr6r\") pod \"payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl\" (UID: \"968de32b-1dea-422a-a1aa-dc4f78412fbb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"
Sep 12 00:56:14.595256 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.595235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r7q8\" (UniqueName: \"kubernetes.io/projected/bdca3a0b-aea6-42f7-8550-4a0f4d23209b-kube-api-access-2r7q8\") pod \"payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs\" (UID: \"bdca3a0b-aea6-42f7-8550-4a0f4d23209b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"
Sep 12 00:56:14.644965 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.644935    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"
Sep 12 00:56:14.666969 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.666939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp" event={"ID":"908baaef-2748-459c-bd57-1f0d994c29bd","Type":"ContainerStarted","Data":"ac8e26db35c5138f1a3955dc76144a8d4591f78c983e1b902ff2715b77141aae"}
Sep 12 00:56:14.667074 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.667001    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp" event={"ID":"908baaef-2748-459c-bd57-1f0d994c29bd","Type":"ContainerStarted","Data":"9113c2a4466b7051d1e8379c0fa9641fe0b7ce21265c275b30a4473574610bf8"}
Sep 12 00:56:14.667333 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.667309    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"
Sep 12 00:56:14.697890 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.697830    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp" podStartSLOduration=1.6978125080000002 podStartE2EDuration="1.697812508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:14.695986718 +0000 UTC m=+799.615278506" watchObservedRunningTime="2026-09-12 00:56:14.697812508 +0000 UTC m=+799.617104294"
Sep 12 00:56:14.700554 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.700098    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"
Sep 12 00:56:14.927076 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.927052    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"]
Sep 12 00:56:14.929404 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:56:14.929379    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdca3a0b_aea6_42f7_8550_4a0f4d23209b.slice/crio-a0afc2bd4078a967d6d6aee419d4016824bc76dd3077d98de570e6e2ce53e28f WatchSource:0}: Error finding container a0afc2bd4078a967d6d6aee419d4016824bc76dd3077d98de570e6e2ce53e28f: Status 404 returned error can't find the container with id a0afc2bd4078a967d6d6aee419d4016824bc76dd3077d98de570e6e2ce53e28f
Sep 12 00:56:14.985177 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:14.985153    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"]
Sep 12 00:56:14.986933 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:56:14.986905    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968de32b_1dea_422a_a1aa_dc4f78412fbb.slice/crio-e0f6ef4a1490c387341392b16883666f28f6dda3d05ada66e2f170c469198e82 WatchSource:0}: Error finding container e0f6ef4a1490c387341392b16883666f28f6dda3d05ada66e2f170c469198e82: Status 404 returned error can't find the container with id e0f6ef4a1490c387341392b16883666f28f6dda3d05ada66e2f170c469198e82
Sep 12 00:56:15.672993 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:15.672957    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs" event={"ID":"bdca3a0b-aea6-42f7-8550-4a0f4d23209b","Type":"ContainerStarted","Data":"110284eadc9c9790603563ba76254ae1b4e4f8cb583fa98d7b390b1926d4000f"}
Sep 12 00:56:15.673448 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:15.673014    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs" event={"ID":"bdca3a0b-aea6-42f7-8550-4a0f4d23209b","Type":"ContainerStarted","Data":"a0afc2bd4078a967d6d6aee419d4016824bc76dd3077d98de570e6e2ce53e28f"}
Sep 12 00:56:15.673448 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:15.673125    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"
Sep 12 00:56:15.674563 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:15.674536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl" event={"ID":"968de32b-1dea-422a-a1aa-dc4f78412fbb","Type":"ContainerStarted","Data":"165045b60efdd6f6e98b91583dd79945c39f5fa757d34ac3766fbfb7b2d48ddd"}
Sep 12 00:56:15.674696 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:15.674586    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl" event={"ID":"968de32b-1dea-422a-a1aa-dc4f78412fbb","Type":"ContainerStarted","Data":"e0f6ef4a1490c387341392b16883666f28f6dda3d05ada66e2f170c469198e82"}
Sep 12 00:56:15.691296 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:15.691252    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs" podStartSLOduration=1.6912414980000001 podStartE2EDuration="1.691241498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:15.690818744 +0000 UTC m=+800.610110531" watchObservedRunningTime="2026-09-12 00:56:15.691241498 +0000 UTC m=+800.610533285"
Sep 12 00:56:15.710503 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:15.710453    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl" podStartSLOduration=1.710437427 podStartE2EDuration="1.710437427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:15.707640151 +0000 UTC m=+800.626931939" watchObservedRunningTime="2026-09-12 00:56:15.710437427 +0000 UTC m=+800.629729211"
Sep 12 00:56:16.680164 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:16.680128    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"
Sep 12 00:56:23.183239 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.183169    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"]
Sep 12 00:56:23.186629 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.186596    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"
Sep 12 00:56:23.189168 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.189149    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-64f698\""
Sep 12 00:56:23.199591 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.199560    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"]
Sep 12 00:56:23.278200 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.278156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41700895-1ced-4f45-a3bb-c21318cd30b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx\" (UID: \"41700895-1ced-4f45-a3bb-c21318cd30b5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"
Sep 12 00:56:23.278363 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.278206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9m49\" (UniqueName: \"kubernetes.io/projected/41700895-1ced-4f45-a3bb-c21318cd30b5-kube-api-access-p9m49\") pod \"payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx\" (UID: \"41700895-1ced-4f45-a3bb-c21318cd30b5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"
Sep 12 00:56:23.333377 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.333339    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"]
Sep 12 00:56:23.338135 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.338103    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"
Sep 12 00:56:23.341718 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.341686    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-64f698-dockercfg-6nwhm\""
Sep 12 00:56:23.356433 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.356399    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"]
Sep 12 00:56:23.380044 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.380003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41700895-1ced-4f45-a3bb-c21318cd30b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx\" (UID: \"41700895-1ced-4f45-a3bb-c21318cd30b5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"
Sep 12 00:56:23.380212 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.380054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9m49\" (UniqueName: \"kubernetes.io/projected/41700895-1ced-4f45-a3bb-c21318cd30b5-kube-api-access-p9m49\") pod \"payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx\" (UID: \"41700895-1ced-4f45-a3bb-c21318cd30b5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"
Sep 12 00:56:23.380212 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.380098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1993dec8-ef1d-4c0b-b1af-2bc5279fe81b-plugins-config-volume\") pod \"payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz\" (UID: \"1993dec8-ef1d-4c0b-b1af-2bc5279fe81b\") " pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"
Sep 12 00:56:23.380212 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.380141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54ll\" (UniqueName: \"kubernetes.io/projected/1993dec8-ef1d-4c0b-b1af-2bc5279fe81b-kube-api-access-c54ll\") pod \"payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz\" (UID: \"1993dec8-ef1d-4c0b-b1af-2bc5279fe81b\") " pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"
Sep 12 00:56:23.380843 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.380821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41700895-1ced-4f45-a3bb-c21318cd30b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx\" (UID: \"41700895-1ced-4f45-a3bb-c21318cd30b5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"
Sep 12 00:56:23.388738 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.388706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9m49\" (UniqueName: \"kubernetes.io/projected/41700895-1ced-4f45-a3bb-c21318cd30b5-kube-api-access-p9m49\") pod \"payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx\" (UID: \"41700895-1ced-4f45-a3bb-c21318cd30b5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"
Sep 12 00:56:23.481565 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.481464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1993dec8-ef1d-4c0b-b1af-2bc5279fe81b-plugins-config-volume\") pod \"payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz\" (UID: \"1993dec8-ef1d-4c0b-b1af-2bc5279fe81b\") " pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"
Sep 12 00:56:23.481565 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.481541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54ll\" (UniqueName: \"kubernetes.io/projected/1993dec8-ef1d-4c0b-b1af-2bc5279fe81b-kube-api-access-c54ll\") pod \"payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz\" (UID: \"1993dec8-ef1d-4c0b-b1af-2bc5279fe81b\") " pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"
Sep 12 00:56:23.482207 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.482185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1993dec8-ef1d-4c0b-b1af-2bc5279fe81b-plugins-config-volume\") pod \"payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz\" (UID: \"1993dec8-ef1d-4c0b-b1af-2bc5279fe81b\") " pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"
Sep 12 00:56:23.497334 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.497315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"
Sep 12 00:56:23.532284 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.532256    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54ll\" (UniqueName: \"kubernetes.io/projected/1993dec8-ef1d-4c0b-b1af-2bc5279fe81b-kube-api-access-c54ll\") pod \"payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz\" (UID: \"1993dec8-ef1d-4c0b-b1af-2bc5279fe81b\") " pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"
Sep 12 00:56:23.659626 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.658793    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"
Sep 12 00:56:23.832626 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:23.826953    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"]
Sep 12 00:56:24.012957 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.009268    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"]
Sep 12 00:56:24.669750 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.669205    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"
Sep 12 00:56:24.670225 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.669885    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"
Sep 12 00:56:24.751624 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.743548    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"]
Sep 12 00:56:24.790621 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.785828    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"]
Sep 12 00:56:24.790621 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.786300    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"
Sep 12 00:56:24.790621 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.789770    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b2dc0e67-dockercfg-qvjd9\""
Sep 12 00:56:24.790621 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.790529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx" event={"ID":"41700895-1ced-4f45-a3bb-c21318cd30b5","Type":"ContainerStarted","Data":"7ddc3ec7890c8acb6014fe8edf8d1e703a1e42852c3ff094af1f13ac0c367167"}
Sep 12 00:56:24.790621 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.790590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx" event={"ID":"41700895-1ced-4f45-a3bb-c21318cd30b5","Type":"ContainerStarted","Data":"bc681ea7b831f579c80387c6b6544c495e702afec45ec709f30301a0458b4269"}
Sep 12 00:56:24.791619 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.791116    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"
Sep 12 00:56:24.791619 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.791201    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b2dc0e67\""
Sep 12 00:56:24.792269 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.792057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6caa3fa0-0819-400c-808f-801e6cc23012-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf\" (UID: \"6caa3fa0-0819-400c-808f-801e6cc23012\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"
Sep 12 00:56:24.792269 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.792117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brc92\" (UniqueName: \"kubernetes.io/projected/6caa3fa0-0819-400c-808f-801e6cc23012-kube-api-access-brc92\") pod \"payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf\" (UID: \"6caa3fa0-0819-400c-808f-801e6cc23012\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"
Sep 12 00:56:24.794311 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.793705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz" event={"ID":"1993dec8-ef1d-4c0b-b1af-2bc5279fe81b","Type":"ContainerStarted","Data":"4a98516e2b0094cc016e71555082e3b7801ab2efb7851e3258a44a0739665bfc"}
Sep 12 00:56:24.794311 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.793751    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz" event={"ID":"1993dec8-ef1d-4c0b-b1af-2bc5279fe81b","Type":"ContainerStarted","Data":"d0cb423fd4de99d3887728df25acf21407c3c1a9f2d418f58951e0cab153da95"}
Sep 12 00:56:24.794311 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.794179    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"
Sep 12 00:56:24.849159 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.849097    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx" podStartSLOduration=1.849060133 podStartE2EDuration="1.849060133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:24.827641016 +0000 UTC m=+809.746932805" watchObservedRunningTime="2026-09-12 00:56:24.849060133 +0000 UTC m=+809.768351920"
Sep 12 00:56:24.893040 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.893002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6caa3fa0-0819-400c-808f-801e6cc23012-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf\" (UID: \"6caa3fa0-0819-400c-808f-801e6cc23012\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"
Sep 12 00:56:24.893217 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.893071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brc92\" (UniqueName: \"kubernetes.io/projected/6caa3fa0-0819-400c-808f-801e6cc23012-kube-api-access-brc92\") pod \"payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf\" (UID: \"6caa3fa0-0819-400c-808f-801e6cc23012\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"
Sep 12 00:56:24.893917 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.893884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6caa3fa0-0819-400c-808f-801e6cc23012-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf\" (UID: \"6caa3fa0-0819-400c-808f-801e6cc23012\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"
Sep 12 00:56:24.922445 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:24.922361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brc92\" (UniqueName: \"kubernetes.io/projected/6caa3fa0-0819-400c-808f-801e6cc23012-kube-api-access-brc92\") pod \"payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf\" (UID: \"6caa3fa0-0819-400c-808f-801e6cc23012\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"
Sep 12 00:56:25.114929 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:25.114671    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"
Sep 12 00:56:25.448292 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:25.448182    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz" podStartSLOduration=2.448158383 podStartE2EDuration="2.448158383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:24.850395484 +0000 UTC m=+809.769687299" watchObservedRunningTime="2026-09-12 00:56:25.448158383 +0000 UTC m=+810.367450172"
Sep 12 00:56:25.450684 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:25.450651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"]
Sep 12 00:56:25.802819 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:25.802379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf" event={"ID":"6caa3fa0-0819-400c-808f-801e6cc23012","Type":"ContainerStarted","Data":"903894fc98a2256181c24067a715669e0a51eca3261e7fe7ea08cb0421041b99"}
Sep 12 00:56:25.802819 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:25.802786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf" event={"ID":"6caa3fa0-0819-400c-808f-801e6cc23012","Type":"ContainerStarted","Data":"dd67d5e49e49d2279475ab44b1eab65ca94c493643824ac20ff3953722280d95"}
Sep 12 00:56:25.803457 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:25.802905    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"
Sep 12 00:56:25.822615 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:25.821663    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf" podStartSLOduration=1.82164409 podStartE2EDuration="1.82164409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:25.819149007 +0000 UTC m=+810.738440794" watchObservedRunningTime="2026-09-12 00:56:25.82164409 +0000 UTC m=+810.740935877"
Sep 12 00:56:26.682069 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:26.682036    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"
Sep 12 00:56:26.682268 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:26.682144    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"
Sep 12 00:56:27.686816 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:27.686783    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"
Sep 12 00:56:28.821651 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:28.821616    2579 generic.go:356] "Generic (PLEG): container finished" podID="171640c8-513b-4b58-a865-08d1cf2faf75" containerID="771af17de8c7920efbfb1df998f77e8819bc1a4618296cd30333f3b9d971f685" exitCode=0
Sep 12 00:56:28.821651 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:28.821628    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc" event={"ID":"171640c8-513b-4b58-a865-08d1cf2faf75","Type":"ContainerDied","Data":"771af17de8c7920efbfb1df998f77e8819bc1a4618296cd30333f3b9d971f685"}
Sep 12 00:56:28.822221 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:28.822194    2579 scope.go:117] "RemoveContainer" containerID="771af17de8c7920efbfb1df998f77e8819bc1a4618296cd30333f3b9d971f685"
Sep 12 00:56:28.823547 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:28.823517    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:56:28.823674 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:28.823565    2579 generic.go:356] "Generic (PLEG): container finished" podID="7dd61045-fc5a-46f4-adf5-50d41b74a5a6" containerID="5bfb7cc88ad20236757040a0d179722cf3f6c65d0ef544c635be3dc7b64c8d18" exitCode=0
Sep 12 00:56:28.823674 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:28.823584    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:56:28.823674 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:28.823631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7" event={"ID":"7dd61045-fc5a-46f4-adf5-50d41b74a5a6","Type":"ContainerDied","Data":"5bfb7cc88ad20236757040a0d179722cf3f6c65d0ef544c635be3dc7b64c8d18"}
Sep 12 00:56:28.823674 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:28.823647    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:56:28.823983 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:28.823969    2579 scope.go:117] "RemoveContainer" containerID="5bfb7cc88ad20236757040a0d179722cf3f6c65d0ef544c635be3dc7b64c8d18"
Sep 12 00:56:28.825057 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:28.825029    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:56:28.825147 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:28.825104    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:56:28.825191 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:28.825151    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:56:29.858529 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:29.858489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc" event={"ID":"171640c8-513b-4b58-a865-08d1cf2faf75","Type":"ContainerStarted","Data":"5c837d175f9d150aab35766308abda123cb0e9e4e228672a754c4f0bb7a408e2"}
Sep 12 00:56:29.861684 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:29.861658    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7" event={"ID":"7dd61045-fc5a-46f4-adf5-50d41b74a5a6","Type":"ContainerStarted","Data":"e453c44dd91bec7f49d41dd257c3942d42ed830e722d54668db9168631b07b55"}
Sep 12 00:56:29.963411 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:29.963375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:29.963626 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:29.963577    2579 patch_prober.go:28] interesting pod/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 12 00:56:29.963696 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:29.963635    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc" podUID="171640c8-513b-4b58-a865-08d1cf2faf75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 12 00:56:30.696192 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:30.696158    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:30.696473 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:30.696446    2579 patch_prober.go:28] interesting pod/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 12 00:56:30.696575 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:30.696497    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7" podUID="7dd61045-fc5a-46f4-adf5-50d41b74a5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 12 00:56:30.964015 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:30.963929    2579 patch_prober.go:28] interesting pod/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 12 00:56:30.964015 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:30.963989    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc" podUID="171640c8-513b-4b58-a865-08d1cf2faf75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 12 00:56:31.696655 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:31.696623    2579 patch_prober.go:28] interesting pod/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 12 00:56:31.696815 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:31.696669    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7" podUID="7dd61045-fc5a-46f4-adf5-50d41b74a5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 12 00:56:31.964547 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:31.964456    2579 patch_prober.go:28] interesting pod/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 12 00:56:31.964547 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:31.964511    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc" podUID="171640c8-513b-4b58-a865-08d1cf2faf75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 12 00:56:32.665097 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.665062    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"]
Sep 12 00:56:32.667435 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.667420    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"
Sep 12 00:56:32.670127 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.670105    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bf547a\""
Sep 12 00:56:32.670379 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.670360    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bf547a-dockercfg-nztbx\""
Sep 12 00:56:32.681902 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.681882    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"]
Sep 12 00:56:32.696155 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.696130    2579 patch_prober.go:28] interesting pod/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 12 00:56:32.696263 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.696182    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7" podUID="7dd61045-fc5a-46f4-adf5-50d41b74a5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 12 00:56:32.785079 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.785043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvzp6\" (UniqueName: \"kubernetes.io/projected/b7852365-9fc3-416e-b626-a9f421d56089-kube-api-access-kvzp6\") pod \"payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889\" (UID: \"b7852365-9fc3-416e-b626-a9f421d56089\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"
Sep 12 00:56:32.785256 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.785094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7852365-9fc3-416e-b626-a9f421d56089-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889\" (UID: \"b7852365-9fc3-416e-b626-a9f421d56089\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"
Sep 12 00:56:32.886540 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.886511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvzp6\" (UniqueName: \"kubernetes.io/projected/b7852365-9fc3-416e-b626-a9f421d56089-kube-api-access-kvzp6\") pod \"payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889\" (UID: \"b7852365-9fc3-416e-b626-a9f421d56089\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"
Sep 12 00:56:32.886736 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.886556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7852365-9fc3-416e-b626-a9f421d56089-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889\" (UID: \"b7852365-9fc3-416e-b626-a9f421d56089\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"
Sep 12 00:56:32.887309 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.887290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7852365-9fc3-416e-b626-a9f421d56089-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889\" (UID: \"b7852365-9fc3-416e-b626-a9f421d56089\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"
Sep 12 00:56:32.894577 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.894545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvzp6\" (UniqueName: \"kubernetes.io/projected/b7852365-9fc3-416e-b626-a9f421d56089-kube-api-access-kvzp6\") pod \"payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889\" (UID: \"b7852365-9fc3-416e-b626-a9f421d56089\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"
Sep 12 00:56:32.964098 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.964021    2579 patch_prober.go:28] interesting pod/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 12 00:56:32.964098 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.964082    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc" podUID="171640c8-513b-4b58-a865-08d1cf2faf75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 12 00:56:32.977558 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:32.977529    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"
Sep 12 00:56:33.355657 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:33.351914    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"]
Sep 12 00:56:33.696379 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:33.696346    2579 patch_prober.go:28] interesting pod/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 12 00:56:33.696519 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:33.696396    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7" podUID="7dd61045-fc5a-46f4-adf5-50d41b74a5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 12 00:56:33.884597 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:33.884559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889" event={"ID":"b7852365-9fc3-416e-b626-a9f421d56089","Type":"ContainerStarted","Data":"ea0e4916e5ad1a8866adff6cdb8fda16ca826563c58bb91e241f228e0d6a1558"}
Sep 12 00:56:33.884781 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:33.884634    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889" event={"ID":"b7852365-9fc3-416e-b626-a9f421d56089","Type":"ContainerStarted","Data":"37fdf94d3234a98c5eb94c6b2e74517ca5a3ca58e22d65c6d133e71454c591b9"}
Sep 12 00:56:33.884883 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:33.884850    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"
Sep 12 00:56:33.964575 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:33.964468    2579 patch_prober.go:28] interesting pod/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 12 00:56:33.964575 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:33.964547    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc" podUID="171640c8-513b-4b58-a865-08d1cf2faf75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 12 00:56:34.696070 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:34.696035    2579 patch_prober.go:28] interesting pod/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body=
Sep 12 00:56:34.696440 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:34.696087    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7" podUID="7dd61045-fc5a-46f4-adf5-50d41b74a5a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused"
Sep 12 00:56:34.964587 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:34.964498    2579 patch_prober.go:28] interesting pod/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Sep 12 00:56:34.964587 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:34.964566    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc" podUID="171640c8-513b-4b58-a865-08d1cf2faf75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Sep 12 00:56:35.699669 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:35.699630    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:35.700127 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:35.699981    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:35.700650 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:35.700634    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 00:56:35.720774 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:35.720725    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889" podStartSLOduration=3.72071036 podStartE2EDuration="3.72071036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:33.903355663 +0000 UTC m=+818.822647463" watchObservedRunningTime="2026-09-12 00:56:35.72071036 +0000 UTC m=+820.640002162"
Sep 12 00:56:35.968458 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:35.968385    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:35.968669 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:35.968647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:35.969375 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:35.969356    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 00:56:36.810474 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:36.810440    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"
Sep 12 00:56:36.810993 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:36.810646    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"
Sep 12 00:56:36.810993 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:36.810704    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"
Sep 12 00:56:44.893159 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:44.893128    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"
Sep 12 00:56:45.486765 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.486728    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"]
Sep 12 00:56:45.486994 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.486943    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf" podUID="6caa3fa0-0819-400c-808f-801e6cc23012" containerName="payload-pre-processing" containerID="cri-o://903894fc98a2256181c24067a715669e0a51eca3261e7fe7ea08cb0421041b99" gracePeriod=30
Sep 12 00:56:45.748080 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.748015    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"
Sep 12 00:56:45.805407 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.805367    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6caa3fa0-0819-400c-808f-801e6cc23012-plugins-config-volume\") pod \"6caa3fa0-0819-400c-808f-801e6cc23012\" (UID: \"6caa3fa0-0819-400c-808f-801e6cc23012\") "
Sep 12 00:56:45.805407 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.805402    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brc92\" (UniqueName: \"kubernetes.io/projected/6caa3fa0-0819-400c-808f-801e6cc23012-kube-api-access-brc92\") pod \"6caa3fa0-0819-400c-808f-801e6cc23012\" (UID: \"6caa3fa0-0819-400c-808f-801e6cc23012\") "
Sep 12 00:56:45.805799 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.805769    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6caa3fa0-0819-400c-808f-801e6cc23012-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6caa3fa0-0819-400c-808f-801e6cc23012" (UID: "6caa3fa0-0819-400c-808f-801e6cc23012"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:56:45.807633 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.807593    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6caa3fa0-0819-400c-808f-801e6cc23012-kube-api-access-brc92" (OuterVolumeSpecName: "kube-api-access-brc92") pod "6caa3fa0-0819-400c-808f-801e6cc23012" (UID: "6caa3fa0-0819-400c-808f-801e6cc23012"). InnerVolumeSpecName "kube-api-access-brc92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:56:45.906910 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.906876    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6caa3fa0-0819-400c-808f-801e6cc23012-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:56:45.906910 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.906903    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brc92\" (UniqueName: \"kubernetes.io/projected/6caa3fa0-0819-400c-808f-801e6cc23012-kube-api-access-brc92\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:56:45.952187 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.952144    2579 generic.go:356] "Generic (PLEG): container finished" podID="6caa3fa0-0819-400c-808f-801e6cc23012" containerID="903894fc98a2256181c24067a715669e0a51eca3261e7fe7ea08cb0421041b99" exitCode=0
Sep 12 00:56:45.952369 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.952201    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"
Sep 12 00:56:45.952369 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.952244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf" event={"ID":"6caa3fa0-0819-400c-808f-801e6cc23012","Type":"ContainerDied","Data":"903894fc98a2256181c24067a715669e0a51eca3261e7fe7ea08cb0421041b99"}
Sep 12 00:56:45.952369 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.952316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf" event={"ID":"6caa3fa0-0819-400c-808f-801e6cc23012","Type":"ContainerDied","Data":"dd67d5e49e49d2279475ab44b1eab65ca94c493643824ac20ff3953722280d95"}
Sep 12 00:56:45.952369 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.952352    2579 scope.go:117] "RemoveContainer" containerID="903894fc98a2256181c24067a715669e0a51eca3261e7fe7ea08cb0421041b99"
Sep 12 00:56:45.966771 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.966649    2579 scope.go:117] "RemoveContainer" containerID="903894fc98a2256181c24067a715669e0a51eca3261e7fe7ea08cb0421041b99"
Sep 12 00:56:45.967182 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:56:45.967154    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"903894fc98a2256181c24067a715669e0a51eca3261e7fe7ea08cb0421041b99\": container with ID starting with 903894fc98a2256181c24067a715669e0a51eca3261e7fe7ea08cb0421041b99 not found: ID does not exist" containerID="903894fc98a2256181c24067a715669e0a51eca3261e7fe7ea08cb0421041b99"
Sep 12 00:56:45.967307 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.967193    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"903894fc98a2256181c24067a715669e0a51eca3261e7fe7ea08cb0421041b99"} err="failed to get container status \"903894fc98a2256181c24067a715669e0a51eca3261e7fe7ea08cb0421041b99\": rpc error: code = NotFound desc = could not find container \"903894fc98a2256181c24067a715669e0a51eca3261e7fe7ea08cb0421041b99\": container with ID starting with 903894fc98a2256181c24067a715669e0a51eca3261e7fe7ea08cb0421041b99 not found: ID does not exist"
Sep 12 00:56:45.978497 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.978462    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"]
Sep 12 00:56:45.983206 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:45.983179    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b2dc0e67-98d9b9d4b-n9wjf"]
Sep 12 00:56:47.590639 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:47.590588    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6caa3fa0-0819-400c-808f-801e6cc23012" path="/var/lib/kubelet/pods/6caa3fa0-0819-400c-808f-801e6cc23012/volumes"
Sep 12 00:56:54.097215 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.094806    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"]
Sep 12 00:56:54.097215 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.095643    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6caa3fa0-0819-400c-808f-801e6cc23012" containerName="payload-pre-processing"
Sep 12 00:56:54.097215 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.095665    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6caa3fa0-0819-400c-808f-801e6cc23012" containerName="payload-pre-processing"
Sep 12 00:56:54.097215 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.095866    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6caa3fa0-0819-400c-808f-801e6cc23012" containerName="payload-pre-processing"
Sep 12 00:56:54.098875 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.098851    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"
Sep 12 00:56:54.102740 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.102721    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f22962b3\""
Sep 12 00:56:54.102850 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.102794    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f22962b3-dockercfg-769jw\""
Sep 12 00:56:54.109648 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.109618    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"]
Sep 12 00:56:54.183116 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.183033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zklhc\" (UniqueName: \"kubernetes.io/projected/29448a4a-5656-4817-b763-c8b6f005e6a5-kube-api-access-zklhc\") pod \"payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8\" (UID: \"29448a4a-5656-4817-b763-c8b6f005e6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"
Sep 12 00:56:54.183483 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.183450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29448a4a-5656-4817-b763-c8b6f005e6a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8\" (UID: \"29448a4a-5656-4817-b763-c8b6f005e6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"
Sep 12 00:56:54.284592 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.284554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29448a4a-5656-4817-b763-c8b6f005e6a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8\" (UID: \"29448a4a-5656-4817-b763-c8b6f005e6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"
Sep 12 00:56:54.284819 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.284666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zklhc\" (UniqueName: \"kubernetes.io/projected/29448a4a-5656-4817-b763-c8b6f005e6a5-kube-api-access-zklhc\") pod \"payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8\" (UID: \"29448a4a-5656-4817-b763-c8b6f005e6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"
Sep 12 00:56:54.285218 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.285198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29448a4a-5656-4817-b763-c8b6f005e6a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8\" (UID: \"29448a4a-5656-4817-b763-c8b6f005e6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"
Sep 12 00:56:54.294374 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.294350    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zklhc\" (UniqueName: \"kubernetes.io/projected/29448a4a-5656-4817-b763-c8b6f005e6a5-kube-api-access-zklhc\") pod \"payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8\" (UID: \"29448a4a-5656-4817-b763-c8b6f005e6a5\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"
Sep 12 00:56:54.411137 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.411042    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"
Sep 12 00:56:54.685581 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:54.685556    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"]
Sep 12 00:56:54.688183 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:56:54.688156    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29448a4a_5656_4817_b763_c8b6f005e6a5.slice/crio-9c8b693aefc92e811268354c9113f244dc829fb540b2affd97d642caf1e68840 WatchSource:0}: Error finding container 9c8b693aefc92e811268354c9113f244dc829fb540b2affd97d642caf1e68840: Status 404 returned error can't find the container with id 9c8b693aefc92e811268354c9113f244dc829fb540b2affd97d642caf1e68840
Sep 12 00:56:55.004450 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:55.004361    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8" event={"ID":"29448a4a-5656-4817-b763-c8b6f005e6a5","Type":"ContainerStarted","Data":"3d4dd985ed5abd148302de2684e03faf7f1df80eca580f354396d99ecfa1edb0"}
Sep 12 00:56:55.004450 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:55.004423    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8" event={"ID":"29448a4a-5656-4817-b763-c8b6f005e6a5","Type":"ContainerStarted","Data":"9c8b693aefc92e811268354c9113f244dc829fb540b2affd97d642caf1e68840"}
Sep 12 00:56:55.004686 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:55.004454    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"
Sep 12 00:56:55.023203 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:56:55.022941    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8" podStartSLOduration=1.022920695 podStartE2EDuration="1.022920695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:56:55.022126946 +0000 UTC m=+839.941418732" watchObservedRunningTime="2026-09-12 00:56:55.022920695 +0000 UTC m=+839.942212482"
Sep 12 00:57:06.011488 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:06.011457    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"
Sep 12 00:57:31.561190 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:31.561155    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"]
Sep 12 00:57:31.561783 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:31.561495    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8" podUID="29448a4a-5656-4817-b763-c8b6f005e6a5" containerName="payload-pre-processing" containerID="cri-o://3d4dd985ed5abd148302de2684e03faf7f1df80eca580f354396d99ecfa1edb0" gracePeriod=30
Sep 12 00:57:31.816544 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:31.816491    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"
Sep 12 00:57:31.906699 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:31.906660    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29448a4a-5656-4817-b763-c8b6f005e6a5-plugins-config-volume\") pod \"29448a4a-5656-4817-b763-c8b6f005e6a5\" (UID: \"29448a4a-5656-4817-b763-c8b6f005e6a5\") "
Sep 12 00:57:31.906860 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:31.906719    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zklhc\" (UniqueName: \"kubernetes.io/projected/29448a4a-5656-4817-b763-c8b6f005e6a5-kube-api-access-zklhc\") pod \"29448a4a-5656-4817-b763-c8b6f005e6a5\" (UID: \"29448a4a-5656-4817-b763-c8b6f005e6a5\") "
Sep 12 00:57:31.907080 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:31.907048    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29448a4a-5656-4817-b763-c8b6f005e6a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29448a4a-5656-4817-b763-c8b6f005e6a5" (UID: "29448a4a-5656-4817-b763-c8b6f005e6a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:57:31.908850 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:31.908825    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29448a4a-5656-4817-b763-c8b6f005e6a5-kube-api-access-zklhc" (OuterVolumeSpecName: "kube-api-access-zklhc") pod "29448a4a-5656-4817-b763-c8b6f005e6a5" (UID: "29448a4a-5656-4817-b763-c8b6f005e6a5"). InnerVolumeSpecName "kube-api-access-zklhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:57:32.007513 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:32.007477    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29448a4a-5656-4817-b763-c8b6f005e6a5-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:57:32.007513 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:32.007506    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zklhc\" (UniqueName: \"kubernetes.io/projected/29448a4a-5656-4817-b763-c8b6f005e6a5-kube-api-access-zklhc\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:57:32.195515 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:32.195478    2579 generic.go:356] "Generic (PLEG): container finished" podID="29448a4a-5656-4817-b763-c8b6f005e6a5" containerID="3d4dd985ed5abd148302de2684e03faf7f1df80eca580f354396d99ecfa1edb0" exitCode=0
Sep 12 00:57:32.195694 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:32.195521    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"
Sep 12 00:57:32.195694 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:32.195525    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8" event={"ID":"29448a4a-5656-4817-b763-c8b6f005e6a5","Type":"ContainerDied","Data":"3d4dd985ed5abd148302de2684e03faf7f1df80eca580f354396d99ecfa1edb0"}
Sep 12 00:57:32.195694 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:32.195580    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8" event={"ID":"29448a4a-5656-4817-b763-c8b6f005e6a5","Type":"ContainerDied","Data":"9c8b693aefc92e811268354c9113f244dc829fb540b2affd97d642caf1e68840"}
Sep 12 00:57:32.195694 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:32.195625    2579 scope.go:117] "RemoveContainer" containerID="3d4dd985ed5abd148302de2684e03faf7f1df80eca580f354396d99ecfa1edb0"
Sep 12 00:57:32.205507 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:32.205489    2579 scope.go:117] "RemoveContainer" containerID="3d4dd985ed5abd148302de2684e03faf7f1df80eca580f354396d99ecfa1edb0"
Sep 12 00:57:32.205779 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:57:32.205760    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4dd985ed5abd148302de2684e03faf7f1df80eca580f354396d99ecfa1edb0\": container with ID starting with 3d4dd985ed5abd148302de2684e03faf7f1df80eca580f354396d99ecfa1edb0 not found: ID does not exist" containerID="3d4dd985ed5abd148302de2684e03faf7f1df80eca580f354396d99ecfa1edb0"
Sep 12 00:57:32.205856 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:32.205787    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4dd985ed5abd148302de2684e03faf7f1df80eca580f354396d99ecfa1edb0"} err="failed to get container status \"3d4dd985ed5abd148302de2684e03faf7f1df80eca580f354396d99ecfa1edb0\": rpc error: code = NotFound desc = could not find container \"3d4dd985ed5abd148302de2684e03faf7f1df80eca580f354396d99ecfa1edb0\": container with ID starting with 3d4dd985ed5abd148302de2684e03faf7f1df80eca580f354396d99ecfa1edb0 not found: ID does not exist"
Sep 12 00:57:32.218318 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:32.218295    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"]
Sep 12 00:57:32.223921 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:32.223897    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f22962b3-6b5f4c6d78-hdlk8"]
Sep 12 00:57:33.590981 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:33.590948    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29448a4a-5656-4817-b763-c8b6f005e6a5" path="/var/lib/kubelet/pods/29448a4a-5656-4817-b763-c8b6f005e6a5/volumes"
Sep 12 00:57:38.188302 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.188270    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"]
Sep 12 00:57:38.188680 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.188658    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29448a4a-5656-4817-b763-c8b6f005e6a5" containerName="payload-pre-processing"
Sep 12 00:57:38.188680 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.188669    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29448a4a-5656-4817-b763-c8b6f005e6a5" containerName="payload-pre-processing"
Sep 12 00:57:38.188760 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.188727    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="29448a4a-5656-4817-b763-c8b6f005e6a5" containerName="payload-pre-processing"
Sep 12 00:57:38.192914 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.192894    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"]
Sep 12 00:57:38.193049 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.193026    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"
Sep 12 00:57:38.195734 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.195713    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6e7edcc4\""
Sep 12 00:57:38.195851 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.195767    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6e7edcc4-dockercfg-xlzmc\""
Sep 12 00:57:38.196074 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.196060    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"
Sep 12 00:57:38.203443 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.203424    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"]
Sep 12 00:57:38.222276 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.222255    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"]
Sep 12 00:57:38.262763 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.262728    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5zwf\" (UniqueName: \"kubernetes.io/projected/36df15ba-d596-4de4-85ec-f2668e3b3f7d-kube-api-access-c5zwf\") pod \"payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm\" (UID: \"36df15ba-d596-4de4-85ec-f2668e3b3f7d\") " pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"
Sep 12 00:57:38.262900 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.262842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36df15ba-d596-4de4-85ec-f2668e3b3f7d-plugins-config-volume\") pod \"payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm\" (UID: \"36df15ba-d596-4de4-85ec-f2668e3b3f7d\") " pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"
Sep 12 00:57:38.262900 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.262873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkp8z\" (UniqueName: \"kubernetes.io/projected/8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e-kube-api-access-hkp8z\") pod \"payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp\" (UID: \"8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"
Sep 12 00:57:38.263009 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.262929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp\" (UID: \"8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"
Sep 12 00:57:38.363990 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.363950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36df15ba-d596-4de4-85ec-f2668e3b3f7d-plugins-config-volume\") pod \"payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm\" (UID: \"36df15ba-d596-4de4-85ec-f2668e3b3f7d\") " pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"
Sep 12 00:57:38.364169 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.364003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkp8z\" (UniqueName: \"kubernetes.io/projected/8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e-kube-api-access-hkp8z\") pod \"payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp\" (UID: \"8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"
Sep 12 00:57:38.364169 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.364066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp\" (UID: \"8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"
Sep 12 00:57:38.364169 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.364113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zwf\" (UniqueName: \"kubernetes.io/projected/36df15ba-d596-4de4-85ec-f2668e3b3f7d-kube-api-access-c5zwf\") pod \"payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm\" (UID: \"36df15ba-d596-4de4-85ec-f2668e3b3f7d\") " pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"
Sep 12 00:57:38.364783 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.364762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36df15ba-d596-4de4-85ec-f2668e3b3f7d-plugins-config-volume\") pod \"payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm\" (UID: \"36df15ba-d596-4de4-85ec-f2668e3b3f7d\") " pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"
Sep 12 00:57:38.364872 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.364788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp\" (UID: \"8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"
Sep 12 00:57:38.373824 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.373793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5zwf\" (UniqueName: \"kubernetes.io/projected/36df15ba-d596-4de4-85ec-f2668e3b3f7d-kube-api-access-c5zwf\") pod \"payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm\" (UID: \"36df15ba-d596-4de4-85ec-f2668e3b3f7d\") " pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"
Sep 12 00:57:38.373926 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.373793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkp8z\" (UniqueName: \"kubernetes.io/projected/8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e-kube-api-access-hkp8z\") pod \"payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp\" (UID: \"8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"
Sep 12 00:57:38.506757 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.506661    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"
Sep 12 00:57:38.513978 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.513956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"
Sep 12 00:57:38.787922 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.787896    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"]
Sep 12 00:57:38.790349 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:57:38.790320    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3da9bc_8114_45a0_8c76_ec6aeebd7c1e.slice/crio-e803e1155bcdcedfcb7a9124549c8a243fb135385ec14374efff75e00029ec96 WatchSource:0}: Error finding container e803e1155bcdcedfcb7a9124549c8a243fb135385ec14374efff75e00029ec96: Status 404 returned error can't find the container with id e803e1155bcdcedfcb7a9124549c8a243fb135385ec14374efff75e00029ec96
Sep 12 00:57:38.791943 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:38.791924    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"]
Sep 12 00:57:38.793492 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:57:38.793466    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36df15ba_d596_4de4_85ec_f2668e3b3f7d.slice/crio-5f1b3301fcc1404f46becf45c88d0d74d17d93dffcfedea6d85b201474d5c2a5 WatchSource:0}: Error finding container 5f1b3301fcc1404f46becf45c88d0d74d17d93dffcfedea6d85b201474d5c2a5: Status 404 returned error can't find the container with id 5f1b3301fcc1404f46becf45c88d0d74d17d93dffcfedea6d85b201474d5c2a5
Sep 12 00:57:39.233740 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:39.233702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp" event={"ID":"8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e","Type":"ContainerStarted","Data":"6f86a056176ce1bae6980b3cf2188f918dd4f323855cd245aceaac12a53e8147"}
Sep 12 00:57:39.234206 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:39.233769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp" event={"ID":"8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e","Type":"ContainerStarted","Data":"e803e1155bcdcedfcb7a9124549c8a243fb135385ec14374efff75e00029ec96"}
Sep 12 00:57:39.234206 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:39.233816    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"
Sep 12 00:57:39.235504 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:39.235478    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm" event={"ID":"36df15ba-d596-4de4-85ec-f2668e3b3f7d","Type":"ContainerStarted","Data":"695a87adc05a383a8427097fc188d7843c002f04877fb40c28d6c4be1535911e"}
Sep 12 00:57:39.235644 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:39.235534    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"
Sep 12 00:57:39.235644 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:39.235559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm" event={"ID":"36df15ba-d596-4de4-85ec-f2668e3b3f7d","Type":"ContainerStarted","Data":"5f1b3301fcc1404f46becf45c88d0d74d17d93dffcfedea6d85b201474d5c2a5"}
Sep 12 00:57:39.249498 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:39.249442    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp" podStartSLOduration=1.249422529 podStartE2EDuration="1.249422529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:57:39.249380141 +0000 UTC m=+884.168671929" watchObservedRunningTime="2026-09-12 00:57:39.249422529 +0000 UTC m=+884.168714317"
Sep 12 00:57:39.267050 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:39.266909    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm" podStartSLOduration=1.266891095 podStartE2EDuration="1.266891095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:57:39.266739019 +0000 UTC m=+884.186030805" watchObservedRunningTime="2026-09-12 00:57:39.266891095 +0000 UTC m=+884.186182882"
Sep 12 00:57:50.242884 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:50.242797    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"
Sep 12 00:57:50.243314 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:50.242900    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"
Sep 12 00:57:55.551908 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:55.551872    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 00:57:55.552438 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:57:55.552419    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 00:58:07.072721 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.072687    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"]
Sep 12 00:58:07.073317 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.072900    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm" podUID="36df15ba-d596-4de4-85ec-f2668e3b3f7d" containerName="payload-processing" containerID="cri-o://695a87adc05a383a8427097fc188d7843c002f04877fb40c28d6c4be1535911e" gracePeriod=30
Sep 12 00:58:07.080338 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.080309    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"]
Sep 12 00:58:07.080902 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.080531    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp" podUID="8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e" containerName="payload-pre-processing" containerID="cri-o://6f86a056176ce1bae6980b3cf2188f918dd4f323855cd245aceaac12a53e8147" gracePeriod=30
Sep 12 00:58:07.335048 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.334986    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"
Sep 12 00:58:07.359226 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.359198    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"
Sep 12 00:58:07.390023 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.389988    2579 generic.go:356] "Generic (PLEG): container finished" podID="8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e" containerID="6f86a056176ce1bae6980b3cf2188f918dd4f323855cd245aceaac12a53e8147" exitCode=0
Sep 12 00:58:07.390170 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.390067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp" event={"ID":"8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e","Type":"ContainerDied","Data":"6f86a056176ce1bae6980b3cf2188f918dd4f323855cd245aceaac12a53e8147"}
Sep 12 00:58:07.390170 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.390139    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp" event={"ID":"8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e","Type":"ContainerDied","Data":"e803e1155bcdcedfcb7a9124549c8a243fb135385ec14374efff75e00029ec96"}
Sep 12 00:58:07.390290 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.390174    2579 scope.go:117] "RemoveContainer" containerID="6f86a056176ce1bae6980b3cf2188f918dd4f323855cd245aceaac12a53e8147"
Sep 12 00:58:07.390382 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.390311    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"
Sep 12 00:58:07.392017 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.391919    2579 generic.go:356] "Generic (PLEG): container finished" podID="36df15ba-d596-4de4-85ec-f2668e3b3f7d" containerID="695a87adc05a383a8427097fc188d7843c002f04877fb40c28d6c4be1535911e" exitCode=0
Sep 12 00:58:07.392017 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.391955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm" event={"ID":"36df15ba-d596-4de4-85ec-f2668e3b3f7d","Type":"ContainerDied","Data":"695a87adc05a383a8427097fc188d7843c002f04877fb40c28d6c4be1535911e"}
Sep 12 00:58:07.392017 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.391986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm" event={"ID":"36df15ba-d596-4de4-85ec-f2668e3b3f7d","Type":"ContainerDied","Data":"5f1b3301fcc1404f46becf45c88d0d74d17d93dffcfedea6d85b201474d5c2a5"}
Sep 12 00:58:07.392178 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.392045    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"
Sep 12 00:58:07.407007 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.406375    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkp8z\" (UniqueName: \"kubernetes.io/projected/8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e-kube-api-access-hkp8z\") pod \"8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e\" (UID: \"8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e\") "
Sep 12 00:58:07.407007 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.406965    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36df15ba-d596-4de4-85ec-f2668e3b3f7d-plugins-config-volume\") pod \"36df15ba-d596-4de4-85ec-f2668e3b3f7d\" (UID: \"36df15ba-d596-4de4-85ec-f2668e3b3f7d\") "
Sep 12 00:58:07.407384 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.407356    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36df15ba-d596-4de4-85ec-f2668e3b3f7d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36df15ba-d596-4de4-85ec-f2668e3b3f7d" (UID: "36df15ba-d596-4de4-85ec-f2668e3b3f7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:58:07.407460 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.407429    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e-plugins-config-volume\") pod \"8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e\" (UID: \"8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e\") "
Sep 12 00:58:07.407916 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.407890    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e" (UID: "8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:58:07.408036 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.407931    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5zwf\" (UniqueName: \"kubernetes.io/projected/36df15ba-d596-4de4-85ec-f2668e3b3f7d-kube-api-access-c5zwf\") pod \"36df15ba-d596-4de4-85ec-f2668e3b3f7d\" (UID: \"36df15ba-d596-4de4-85ec-f2668e3b3f7d\") "
Sep 12 00:58:07.408303 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.408285    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36df15ba-d596-4de4-85ec-f2668e3b3f7d-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:07.408376 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.408320    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:07.410691 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.410668    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e-kube-api-access-hkp8z" (OuterVolumeSpecName: "kube-api-access-hkp8z") pod "8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e" (UID: "8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e"). InnerVolumeSpecName "kube-api-access-hkp8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:58:07.412746 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.412711    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36df15ba-d596-4de4-85ec-f2668e3b3f7d-kube-api-access-c5zwf" (OuterVolumeSpecName: "kube-api-access-c5zwf") pod "36df15ba-d596-4de4-85ec-f2668e3b3f7d" (UID: "36df15ba-d596-4de4-85ec-f2668e3b3f7d"). InnerVolumeSpecName "kube-api-access-c5zwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:58:07.418423 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.417887    2579 scope.go:117] "RemoveContainer" containerID="6f86a056176ce1bae6980b3cf2188f918dd4f323855cd245aceaac12a53e8147"
Sep 12 00:58:07.418821 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:58:07.418551    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f86a056176ce1bae6980b3cf2188f918dd4f323855cd245aceaac12a53e8147\": container with ID starting with 6f86a056176ce1bae6980b3cf2188f918dd4f323855cd245aceaac12a53e8147 not found: ID does not exist" containerID="6f86a056176ce1bae6980b3cf2188f918dd4f323855cd245aceaac12a53e8147"
Sep 12 00:58:07.419590 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.418770    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f86a056176ce1bae6980b3cf2188f918dd4f323855cd245aceaac12a53e8147"} err="failed to get container status \"6f86a056176ce1bae6980b3cf2188f918dd4f323855cd245aceaac12a53e8147\": rpc error: code = NotFound desc = could not find container \"6f86a056176ce1bae6980b3cf2188f918dd4f323855cd245aceaac12a53e8147\": container with ID starting with 6f86a056176ce1bae6980b3cf2188f918dd4f323855cd245aceaac12a53e8147 not found: ID does not exist"
Sep 12 00:58:07.419590 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.418952    2579 scope.go:117] "RemoveContainer" containerID="695a87adc05a383a8427097fc188d7843c002f04877fb40c28d6c4be1535911e"
Sep 12 00:58:07.435402 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.435369    2579 scope.go:117] "RemoveContainer" containerID="695a87adc05a383a8427097fc188d7843c002f04877fb40c28d6c4be1535911e"
Sep 12 00:58:07.436187 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:58:07.436057    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695a87adc05a383a8427097fc188d7843c002f04877fb40c28d6c4be1535911e\": container with ID starting with 695a87adc05a383a8427097fc188d7843c002f04877fb40c28d6c4be1535911e not found: ID does not exist" containerID="695a87adc05a383a8427097fc188d7843c002f04877fb40c28d6c4be1535911e"
Sep 12 00:58:07.436187 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.436128    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695a87adc05a383a8427097fc188d7843c002f04877fb40c28d6c4be1535911e"} err="failed to get container status \"695a87adc05a383a8427097fc188d7843c002f04877fb40c28d6c4be1535911e\": rpc error: code = NotFound desc = could not find container \"695a87adc05a383a8427097fc188d7843c002f04877fb40c28d6c4be1535911e\": container with ID starting with 695a87adc05a383a8427097fc188d7843c002f04877fb40c28d6c4be1535911e not found: ID does not exist"
Sep 12 00:58:07.509901 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.509865    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5zwf\" (UniqueName: \"kubernetes.io/projected/36df15ba-d596-4de4-85ec-f2668e3b3f7d-kube-api-access-c5zwf\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:07.509901 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.509903    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkp8z\" (UniqueName: \"kubernetes.io/projected/8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e-kube-api-access-hkp8z\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:07.708494 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.708466    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"]
Sep 12 00:58:07.713699 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.713676    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e7edcc4-cb497bc8b-q8hjp"]
Sep 12 00:58:07.724368 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.724348    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"]
Sep 12 00:58:07.731512 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:07.731491    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e7edcc4-7c56f5bd49-bshmm"]
Sep 12 00:58:09.591659 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:09.591617    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36df15ba-d596-4de4-85ec-f2668e3b3f7d" path="/var/lib/kubelet/pods/36df15ba-d596-4de4-85ec-f2668e3b3f7d/volumes"
Sep 12 00:58:09.592024 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:09.591971    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e" path="/var/lib/kubelet/pods/8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e/volumes"
Sep 12 00:58:13.469905 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.469871    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"]
Sep 12 00:58:13.470264 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.470244    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e" containerName="payload-pre-processing"
Sep 12 00:58:13.470264 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.470257    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e" containerName="payload-pre-processing"
Sep 12 00:58:13.470354 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.470280    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36df15ba-d596-4de4-85ec-f2668e3b3f7d" containerName="payload-processing"
Sep 12 00:58:13.470354 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.470286    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="36df15ba-d596-4de4-85ec-f2668e3b3f7d" containerName="payload-processing"
Sep 12 00:58:13.470354 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.470347    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3da9bc-8114-45a0-8c76-ec6aeebd7c1e" containerName="payload-pre-processing"
Sep 12 00:58:13.470447 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.470357    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="36df15ba-d596-4de4-85ec-f2668e3b3f7d" containerName="payload-processing"
Sep 12 00:58:13.474919 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.474901    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"
Sep 12 00:58:13.477251 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.477226    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-7c4848b5\""
Sep 12 00:58:13.484709 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.484685    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"]
Sep 12 00:58:13.506054 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.506031    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"]
Sep 12 00:58:13.509334 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.509316    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"
Sep 12 00:58:13.511893 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.511876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-7c4848b5-dockercfg-lr8bw\""
Sep 12 00:58:13.524889 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.524864    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"]
Sep 12 00:58:13.561939 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.561915    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xgn9\" (UniqueName: \"kubernetes.io/projected/e39cf356-e81d-410c-b608-eccde607ee3c-kube-api-access-8xgn9\") pod \"payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww\" (UID: \"e39cf356-e81d-410c-b608-eccde607ee3c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"
Sep 12 00:58:13.562051 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.561964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k65nx\" (UniqueName: \"kubernetes.io/projected/8d18c97a-bb56-493c-aafa-92b792029e38-kube-api-access-k65nx\") pod \"payload-processing-e2e-derive-7c4848b5-97495458f-796ft\" (UID: \"8d18c97a-bb56-493c-aafa-92b792029e38\") " pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"
Sep 12 00:58:13.562097 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.562048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d18c97a-bb56-493c-aafa-92b792029e38-plugins-config-volume\") pod \"payload-processing-e2e-derive-7c4848b5-97495458f-796ft\" (UID: \"8d18c97a-bb56-493c-aafa-92b792029e38\") " pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"
Sep 12 00:58:13.562132 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.562106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39cf356-e81d-410c-b608-eccde607ee3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww\" (UID: \"e39cf356-e81d-410c-b608-eccde607ee3c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"
Sep 12 00:58:13.663070 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.663033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39cf356-e81d-410c-b608-eccde607ee3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww\" (UID: \"e39cf356-e81d-410c-b608-eccde607ee3c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"
Sep 12 00:58:13.663070 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.663073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xgn9\" (UniqueName: \"kubernetes.io/projected/e39cf356-e81d-410c-b608-eccde607ee3c-kube-api-access-8xgn9\") pod \"payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww\" (UID: \"e39cf356-e81d-410c-b608-eccde607ee3c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"
Sep 12 00:58:13.663330 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.663121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k65nx\" (UniqueName: \"kubernetes.io/projected/8d18c97a-bb56-493c-aafa-92b792029e38-kube-api-access-k65nx\") pod \"payload-processing-e2e-derive-7c4848b5-97495458f-796ft\" (UID: \"8d18c97a-bb56-493c-aafa-92b792029e38\") " pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"
Sep 12 00:58:13.663330 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.663150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d18c97a-bb56-493c-aafa-92b792029e38-plugins-config-volume\") pod \"payload-processing-e2e-derive-7c4848b5-97495458f-796ft\" (UID: \"8d18c97a-bb56-493c-aafa-92b792029e38\") " pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"
Sep 12 00:58:13.663799 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.663780    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39cf356-e81d-410c-b608-eccde607ee3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww\" (UID: \"e39cf356-e81d-410c-b608-eccde607ee3c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"
Sep 12 00:58:13.663886 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.663868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d18c97a-bb56-493c-aafa-92b792029e38-plugins-config-volume\") pod \"payload-processing-e2e-derive-7c4848b5-97495458f-796ft\" (UID: \"8d18c97a-bb56-493c-aafa-92b792029e38\") " pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"
Sep 12 00:58:13.671670 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.671640    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k65nx\" (UniqueName: \"kubernetes.io/projected/8d18c97a-bb56-493c-aafa-92b792029e38-kube-api-access-k65nx\") pod \"payload-processing-e2e-derive-7c4848b5-97495458f-796ft\" (UID: \"8d18c97a-bb56-493c-aafa-92b792029e38\") " pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"
Sep 12 00:58:13.671769 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.671644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xgn9\" (UniqueName: \"kubernetes.io/projected/e39cf356-e81d-410c-b608-eccde607ee3c-kube-api-access-8xgn9\") pod \"payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww\" (UID: \"e39cf356-e81d-410c-b608-eccde607ee3c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"
Sep 12 00:58:13.786795 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.786711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"
Sep 12 00:58:13.821022 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:13.820987    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"
Sep 12 00:58:14.068134 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:14.068108    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"]
Sep 12 00:58:14.069669 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:58:14.069639    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39cf356_e81d_410c_b608_eccde607ee3c.slice/crio-49c041fd7b9acb3e36134fec91ac8dc8df9180d35ee836739512ce145d745b18 WatchSource:0}: Error finding container 49c041fd7b9acb3e36134fec91ac8dc8df9180d35ee836739512ce145d745b18: Status 404 returned error can't find the container with id 49c041fd7b9acb3e36134fec91ac8dc8df9180d35ee836739512ce145d745b18
Sep 12 00:58:14.118382 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:14.118355    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"]
Sep 12 00:58:14.119559 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:58:14.119521    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d18c97a_bb56_493c_aafa_92b792029e38.slice/crio-288e2bd7c6599c882e2b5258d578fbc28968603963859fa9b02e3216cf83cc22 WatchSource:0}: Error finding container 288e2bd7c6599c882e2b5258d578fbc28968603963859fa9b02e3216cf83cc22: Status 404 returned error can't find the container with id 288e2bd7c6599c882e2b5258d578fbc28968603963859fa9b02e3216cf83cc22
Sep 12 00:58:14.435622 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:14.435559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww" event={"ID":"e39cf356-e81d-410c-b608-eccde607ee3c","Type":"ContainerStarted","Data":"ebe6e34a98840d9786a213dc68c139399a5964bbda8d77bf369b5e535b344a18"}
Sep 12 00:58:14.435839 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:14.435667    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww" event={"ID":"e39cf356-e81d-410c-b608-eccde607ee3c","Type":"ContainerStarted","Data":"49c041fd7b9acb3e36134fec91ac8dc8df9180d35ee836739512ce145d745b18"}
Sep 12 00:58:14.436396 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:14.436373    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"
Sep 12 00:58:14.438592 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:14.438444    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft" event={"ID":"8d18c97a-bb56-493c-aafa-92b792029e38","Type":"ContainerStarted","Data":"476820bad4e356ffe5b39ddd01a3975d8756c60fe168af76b5cb349d484c1ce2"}
Sep 12 00:58:14.438592 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:14.438497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft" event={"ID":"8d18c97a-bb56-493c-aafa-92b792029e38","Type":"ContainerStarted","Data":"288e2bd7c6599c882e2b5258d578fbc28968603963859fa9b02e3216cf83cc22"}
Sep 12 00:58:14.438592 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:14.438553    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"
Sep 12 00:58:14.452135 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:14.452091    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww" podStartSLOduration=1.452073717 podStartE2EDuration="1.452073717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:58:14.451724025 +0000 UTC m=+919.371015817" watchObservedRunningTime="2026-09-12 00:58:14.452073717 +0000 UTC m=+919.371365506"
Sep 12 00:58:14.468479 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:14.468441    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft" podStartSLOduration=1.468428739 podStartE2EDuration="1.468428739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:58:14.467546073 +0000 UTC m=+919.386837877" watchObservedRunningTime="2026-09-12 00:58:14.468428739 +0000 UTC m=+919.387720553"
Sep 12 00:58:25.445698 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:25.445666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"
Sep 12 00:58:26.451332 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:26.451299    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"
Sep 12 00:58:43.338015 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.337969    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"]
Sep 12 00:58:43.338949 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.338910    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft" podUID="8d18c97a-bb56-493c-aafa-92b792029e38" containerName="payload-processing" containerID="cri-o://476820bad4e356ffe5b39ddd01a3975d8756c60fe168af76b5cb349d484c1ce2" gracePeriod=30
Sep 12 00:58:43.341769 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.340626    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"]
Sep 12 00:58:43.341985 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.341955    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww" podUID="e39cf356-e81d-410c-b608-eccde607ee3c" containerName="payload-pre-processing" containerID="cri-o://ebe6e34a98840d9786a213dc68c139399a5964bbda8d77bf369b5e535b344a18" gracePeriod=30
Sep 12 00:58:43.592440 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.592403    2579 generic.go:356] "Generic (PLEG): container finished" podID="e39cf356-e81d-410c-b608-eccde607ee3c" containerID="ebe6e34a98840d9786a213dc68c139399a5964bbda8d77bf369b5e535b344a18" exitCode=0
Sep 12 00:58:43.594652 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.594594    2579 generic.go:356] "Generic (PLEG): container finished" podID="8d18c97a-bb56-493c-aafa-92b792029e38" containerID="476820bad4e356ffe5b39ddd01a3975d8756c60fe168af76b5cb349d484c1ce2" exitCode=0
Sep 12 00:58:43.595213 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.595186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww" event={"ID":"e39cf356-e81d-410c-b608-eccde607ee3c","Type":"ContainerDied","Data":"ebe6e34a98840d9786a213dc68c139399a5964bbda8d77bf369b5e535b344a18"}
Sep 12 00:58:43.595282 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.595252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft" event={"ID":"8d18c97a-bb56-493c-aafa-92b792029e38","Type":"ContainerDied","Data":"476820bad4e356ffe5b39ddd01a3975d8756c60fe168af76b5cb349d484c1ce2"}
Sep 12 00:58:43.620920 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.620142    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"
Sep 12 00:58:43.651233 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.651205    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"
Sep 12 00:58:43.728346 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.728315    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k65nx\" (UniqueName: \"kubernetes.io/projected/8d18c97a-bb56-493c-aafa-92b792029e38-kube-api-access-k65nx\") pod \"8d18c97a-bb56-493c-aafa-92b792029e38\" (UID: \"8d18c97a-bb56-493c-aafa-92b792029e38\") "
Sep 12 00:58:43.728500 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.728354    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d18c97a-bb56-493c-aafa-92b792029e38-plugins-config-volume\") pod \"8d18c97a-bb56-493c-aafa-92b792029e38\" (UID: \"8d18c97a-bb56-493c-aafa-92b792029e38\") "
Sep 12 00:58:43.728500 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.728408    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xgn9\" (UniqueName: \"kubernetes.io/projected/e39cf356-e81d-410c-b608-eccde607ee3c-kube-api-access-8xgn9\") pod \"e39cf356-e81d-410c-b608-eccde607ee3c\" (UID: \"e39cf356-e81d-410c-b608-eccde607ee3c\") "
Sep 12 00:58:43.728500 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.728444    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39cf356-e81d-410c-b608-eccde607ee3c-plugins-config-volume\") pod \"e39cf356-e81d-410c-b608-eccde607ee3c\" (UID: \"e39cf356-e81d-410c-b608-eccde607ee3c\") "
Sep 12 00:58:43.728863 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.728828    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d18c97a-bb56-493c-aafa-92b792029e38-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d18c97a-bb56-493c-aafa-92b792029e38" (UID: "8d18c97a-bb56-493c-aafa-92b792029e38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:58:43.728863 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.728855    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39cf356-e81d-410c-b608-eccde607ee3c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e39cf356-e81d-410c-b608-eccde607ee3c" (UID: "e39cf356-e81d-410c-b608-eccde607ee3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:58:43.730339 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.730319    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d18c97a-bb56-493c-aafa-92b792029e38-kube-api-access-k65nx" (OuterVolumeSpecName: "kube-api-access-k65nx") pod "8d18c97a-bb56-493c-aafa-92b792029e38" (UID: "8d18c97a-bb56-493c-aafa-92b792029e38"). InnerVolumeSpecName "kube-api-access-k65nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:58:43.730419 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.730382    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39cf356-e81d-410c-b608-eccde607ee3c-kube-api-access-8xgn9" (OuterVolumeSpecName: "kube-api-access-8xgn9") pod "e39cf356-e81d-410c-b608-eccde607ee3c" (UID: "e39cf356-e81d-410c-b608-eccde607ee3c"). InnerVolumeSpecName "kube-api-access-8xgn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:58:43.829432 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.829356    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k65nx\" (UniqueName: \"kubernetes.io/projected/8d18c97a-bb56-493c-aafa-92b792029e38-kube-api-access-k65nx\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:43.829432 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.829384    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d18c97a-bb56-493c-aafa-92b792029e38-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:43.829432 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.829393    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xgn9\" (UniqueName: \"kubernetes.io/projected/e39cf356-e81d-410c-b608-eccde607ee3c-kube-api-access-8xgn9\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:43.829432 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:43.829402    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e39cf356-e81d-410c-b608-eccde607ee3c-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:44.601515 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:44.601480    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"
Sep 12 00:58:44.601515 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:44.601500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft" event={"ID":"8d18c97a-bb56-493c-aafa-92b792029e38","Type":"ContainerDied","Data":"288e2bd7c6599c882e2b5258d578fbc28968603963859fa9b02e3216cf83cc22"}
Sep 12 00:58:44.602043 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:44.601563    2579 scope.go:117] "RemoveContainer" containerID="476820bad4e356ffe5b39ddd01a3975d8756c60fe168af76b5cb349d484c1ce2"
Sep 12 00:58:44.603326 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:44.603300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww" event={"ID":"e39cf356-e81d-410c-b608-eccde607ee3c","Type":"ContainerDied","Data":"49c041fd7b9acb3e36134fec91ac8dc8df9180d35ee836739512ce145d745b18"}
Sep 12 00:58:44.603456 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:44.603340    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"
Sep 12 00:58:44.612166 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:44.612147    2579 scope.go:117] "RemoveContainer" containerID="ebe6e34a98840d9786a213dc68c139399a5964bbda8d77bf369b5e535b344a18"
Sep 12 00:58:44.627700 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:44.627676    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"]
Sep 12 00:58:44.632315 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:44.632292    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7c4848b5-97495458f-796ft"]
Sep 12 00:58:44.649863 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:44.649840    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"]
Sep 12 00:58:44.655078 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:44.655053    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7c4848b5-65d8f5576b-mflww"]
Sep 12 00:58:45.591971 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:45.591932    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d18c97a-bb56-493c-aafa-92b792029e38" path="/var/lib/kubelet/pods/8d18c97a-bb56-493c-aafa-92b792029e38/volumes"
Sep 12 00:58:45.592364 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:45.592345    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e39cf356-e81d-410c-b608-eccde607ee3c" path="/var/lib/kubelet/pods/e39cf356-e81d-410c-b608-eccde607ee3c/volumes"
Sep 12 00:58:46.618076 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:46.618042    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-64f698-8ltnb"]
Sep 12 00:58:46.618637 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:46.618595    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e39cf356-e81d-410c-b608-eccde607ee3c" containerName="payload-pre-processing"
Sep 12 00:58:46.618637 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:46.618634    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39cf356-e81d-410c-b608-eccde607ee3c" containerName="payload-pre-processing"
Sep 12 00:58:46.618780 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:46.618660    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d18c97a-bb56-493c-aafa-92b792029e38" containerName="payload-processing"
Sep 12 00:58:46.618780 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:46.618668    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d18c97a-bb56-493c-aafa-92b792029e38" containerName="payload-processing"
Sep 12 00:58:46.618780 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:46.618778    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e39cf356-e81d-410c-b608-eccde607ee3c" containerName="payload-pre-processing"
Sep 12 00:58:46.618927 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:46.618794    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d18c97a-bb56-493c-aafa-92b792029e38" containerName="payload-processing"
Sep 12 00:58:46.623299 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:46.623279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-64f698-8ltnb"
Sep 12 00:58:46.625926 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:46.625898    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 00:58:46.626193 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:46.626174    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2rn9k\""
Sep 12 00:58:46.628411 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:46.628390    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-64f698-8ltnb"]
Sep 12 00:58:46.755584 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:46.755551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73abcfce-3605-4c23-9e39-590c040f4ea3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-64f698-8ltnb\" (UID: \"73abcfce-3605-4c23-9e39-590c040f4ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-64f698-8ltnb"
Sep 12 00:58:46.856208 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:46.856172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73abcfce-3605-4c23-9e39-590c040f4ea3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-64f698-8ltnb\" (UID: \"73abcfce-3605-4c23-9e39-590c040f4ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-64f698-8ltnb"
Sep 12 00:58:46.856801 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:46.856781    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73abcfce-3605-4c23-9e39-590c040f4ea3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-64f698-8ltnb\" (UID: \"73abcfce-3605-4c23-9e39-590c040f4ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-64f698-8ltnb"
Sep 12 00:58:46.934717 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:46.934684    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-64f698-8ltnb"
Sep 12 00:58:47.229306 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:47.229283    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-64f698-8ltnb"]
Sep 12 00:58:47.230788 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:58:47.230761    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73abcfce_3605_4c23_9e39_590c040f4ea3.slice/crio-dc65e7152a7080dce90e347b5238fb48c803d8cacfdbd03af9448ff5d71cd455 WatchSource:0}: Error finding container dc65e7152a7080dce90e347b5238fb48c803d8cacfdbd03af9448ff5d71cd455: Status 404 returned error can't find the container with id dc65e7152a7080dce90e347b5238fb48c803d8cacfdbd03af9448ff5d71cd455
Sep 12 00:58:47.232702 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:47.232687    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 00:58:47.623015 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:47.622936    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-64f698-8ltnb" event={"ID":"73abcfce-3605-4c23-9e39-590c040f4ea3","Type":"ContainerStarted","Data":"dc65e7152a7080dce90e347b5238fb48c803d8cacfdbd03af9448ff5d71cd455"}
Sep 12 00:58:50.640850 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:50.640759    2579 generic.go:356] "Generic (PLEG): container finished" podID="73abcfce-3605-4c23-9e39-590c040f4ea3" containerID="0583e10bb6f7eabc7ca4ed6d9fe344cd0571cc06d7b79022584541a8bf4b2143" exitCode=0
Sep 12 00:58:50.641248 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:50.640845    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-64f698-8ltnb" event={"ID":"73abcfce-3605-4c23-9e39-590c040f4ea3","Type":"ContainerDied","Data":"0583e10bb6f7eabc7ca4ed6d9fe344cd0571cc06d7b79022584541a8bf4b2143"}
Sep 12 00:58:51.780031 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:51.780009    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-64f698-8ltnb"
Sep 12 00:58:51.907003 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:51.906901    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73abcfce-3605-4c23-9e39-590c040f4ea3-maas-api-service-ca\") pod \"73abcfce-3605-4c23-9e39-590c040f4ea3\" (UID: \"73abcfce-3605-4c23-9e39-590c040f4ea3\") "
Sep 12 00:58:51.907349 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:51.907314    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73abcfce-3605-4c23-9e39-590c040f4ea3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73abcfce-3605-4c23-9e39-590c040f4ea3" (UID: "73abcfce-3605-4c23-9e39-590c040f4ea3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:58:52.007973 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:52.007942    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73abcfce-3605-4c23-9e39-590c040f4ea3-maas-api-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:52.651253 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:52.651224    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-64f698-8ltnb"
Sep 12 00:58:52.651403 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:52.651263    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-64f698-8ltnb" event={"ID":"73abcfce-3605-4c23-9e39-590c040f4ea3","Type":"ContainerDied","Data":"dc65e7152a7080dce90e347b5238fb48c803d8cacfdbd03af9448ff5d71cd455"}
Sep 12 00:58:52.651403 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:52.651310    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc65e7152a7080dce90e347b5238fb48c803d8cacfdbd03af9448ff5d71cd455"
Sep 12 00:58:57.180681 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.179351    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"]
Sep 12 00:58:57.180681 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.179464    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"]
Sep 12 00:58:57.180681 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.179724    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx" podUID="41700895-1ced-4f45-a3bb-c21318cd30b5" containerName="payload-pre-processing" containerID="cri-o://7ddc3ec7890c8acb6014fe8edf8d1e703a1e42852c3ff094af1f13ac0c367167" gracePeriod=30
Sep 12 00:58:57.180681 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.180029    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz" podUID="1993dec8-ef1d-4c0b-b1af-2bc5279fe81b" containerName="payload-processing" containerID="cri-o://4a98516e2b0094cc016e71555082e3b7801ab2efb7851e3258a44a0739665bfc" gracePeriod=30
Sep 12 00:58:57.385092 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.385061    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-64f698-8ltnb"]
Sep 12 00:58:57.392122 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.392070    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-64f698-8ltnb"]
Sep 12 00:58:57.474366 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.474341    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"
Sep 12 00:58:57.477180 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.477156    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"
Sep 12 00:58:57.554670 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.554636    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c54ll\" (UniqueName: \"kubernetes.io/projected/1993dec8-ef1d-4c0b-b1af-2bc5279fe81b-kube-api-access-c54ll\") pod \"1993dec8-ef1d-4c0b-b1af-2bc5279fe81b\" (UID: \"1993dec8-ef1d-4c0b-b1af-2bc5279fe81b\") "
Sep 12 00:58:57.554843 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.554687    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1993dec8-ef1d-4c0b-b1af-2bc5279fe81b-plugins-config-volume\") pod \"1993dec8-ef1d-4c0b-b1af-2bc5279fe81b\" (UID: \"1993dec8-ef1d-4c0b-b1af-2bc5279fe81b\") "
Sep 12 00:58:57.554843 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.554726    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9m49\" (UniqueName: \"kubernetes.io/projected/41700895-1ced-4f45-a3bb-c21318cd30b5-kube-api-access-p9m49\") pod \"41700895-1ced-4f45-a3bb-c21318cd30b5\" (UID: \"41700895-1ced-4f45-a3bb-c21318cd30b5\") "
Sep 12 00:58:57.554843 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.554765    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41700895-1ced-4f45-a3bb-c21318cd30b5-plugins-config-volume\") pod \"41700895-1ced-4f45-a3bb-c21318cd30b5\" (UID: \"41700895-1ced-4f45-a3bb-c21318cd30b5\") "
Sep 12 00:58:57.555202 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.555163    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1993dec8-ef1d-4c0b-b1af-2bc5279fe81b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1993dec8-ef1d-4c0b-b1af-2bc5279fe81b" (UID: "1993dec8-ef1d-4c0b-b1af-2bc5279fe81b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:58:57.555330 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.555193    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41700895-1ced-4f45-a3bb-c21318cd30b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41700895-1ced-4f45-a3bb-c21318cd30b5" (UID: "41700895-1ced-4f45-a3bb-c21318cd30b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 00:58:57.556864 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.556841    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41700895-1ced-4f45-a3bb-c21318cd30b5-kube-api-access-p9m49" (OuterVolumeSpecName: "kube-api-access-p9m49") pod "41700895-1ced-4f45-a3bb-c21318cd30b5" (UID: "41700895-1ced-4f45-a3bb-c21318cd30b5"). InnerVolumeSpecName "kube-api-access-p9m49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:58:57.556969 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.556918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1993dec8-ef1d-4c0b-b1af-2bc5279fe81b-kube-api-access-c54ll" (OuterVolumeSpecName: "kube-api-access-c54ll") pod "1993dec8-ef1d-4c0b-b1af-2bc5279fe81b" (UID: "1993dec8-ef1d-4c0b-b1af-2bc5279fe81b"). InnerVolumeSpecName "kube-api-access-c54ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 00:58:57.590680 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.590655    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73abcfce-3605-4c23-9e39-590c040f4ea3" path="/var/lib/kubelet/pods/73abcfce-3605-4c23-9e39-590c040f4ea3/volumes"
Sep 12 00:58:57.655786 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.655758    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c54ll\" (UniqueName: \"kubernetes.io/projected/1993dec8-ef1d-4c0b-b1af-2bc5279fe81b-kube-api-access-c54ll\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:57.655786 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.655782    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1993dec8-ef1d-4c0b-b1af-2bc5279fe81b-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:57.655786 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.655792    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9m49\" (UniqueName: \"kubernetes.io/projected/41700895-1ced-4f45-a3bb-c21318cd30b5-kube-api-access-p9m49\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:57.655996 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.655800    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41700895-1ced-4f45-a3bb-c21318cd30b5-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 00:58:57.677670 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.677640    2579 generic.go:356] "Generic (PLEG): container finished" podID="1993dec8-ef1d-4c0b-b1af-2bc5279fe81b" containerID="4a98516e2b0094cc016e71555082e3b7801ab2efb7851e3258a44a0739665bfc" exitCode=0
Sep 12 00:58:57.677809 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.677685    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"
Sep 12 00:58:57.677809 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.677722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz" event={"ID":"1993dec8-ef1d-4c0b-b1af-2bc5279fe81b","Type":"ContainerDied","Data":"4a98516e2b0094cc016e71555082e3b7801ab2efb7851e3258a44a0739665bfc"}
Sep 12 00:58:57.677809 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.677774    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz" event={"ID":"1993dec8-ef1d-4c0b-b1af-2bc5279fe81b","Type":"ContainerDied","Data":"d0cb423fd4de99d3887728df25acf21407c3c1a9f2d418f58951e0cab153da95"}
Sep 12 00:58:57.677809 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.677795    2579 scope.go:117] "RemoveContainer" containerID="4a98516e2b0094cc016e71555082e3b7801ab2efb7851e3258a44a0739665bfc"
Sep 12 00:58:57.679122 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.679093    2579 generic.go:356] "Generic (PLEG): container finished" podID="41700895-1ced-4f45-a3bb-c21318cd30b5" containerID="7ddc3ec7890c8acb6014fe8edf8d1e703a1e42852c3ff094af1f13ac0c367167" exitCode=0
Sep 12 00:58:57.679226 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.679142    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"
Sep 12 00:58:57.679226 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.679148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx" event={"ID":"41700895-1ced-4f45-a3bb-c21318cd30b5","Type":"ContainerDied","Data":"7ddc3ec7890c8acb6014fe8edf8d1e703a1e42852c3ff094af1f13ac0c367167"}
Sep 12 00:58:57.679226 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.679181    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx" event={"ID":"41700895-1ced-4f45-a3bb-c21318cd30b5","Type":"ContainerDied","Data":"bc681ea7b831f579c80387c6b6544c495e702afec45ec709f30301a0458b4269"}
Sep 12 00:58:57.686779 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.686763    2579 scope.go:117] "RemoveContainer" containerID="4a98516e2b0094cc016e71555082e3b7801ab2efb7851e3258a44a0739665bfc"
Sep 12 00:58:57.686998 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:58:57.686983    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a98516e2b0094cc016e71555082e3b7801ab2efb7851e3258a44a0739665bfc\": container with ID starting with 4a98516e2b0094cc016e71555082e3b7801ab2efb7851e3258a44a0739665bfc not found: ID does not exist" containerID="4a98516e2b0094cc016e71555082e3b7801ab2efb7851e3258a44a0739665bfc"
Sep 12 00:58:57.687048 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.687005    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a98516e2b0094cc016e71555082e3b7801ab2efb7851e3258a44a0739665bfc"} err="failed to get container status \"4a98516e2b0094cc016e71555082e3b7801ab2efb7851e3258a44a0739665bfc\": rpc error: code = NotFound desc = could not find container \"4a98516e2b0094cc016e71555082e3b7801ab2efb7851e3258a44a0739665bfc\": container with ID starting with 4a98516e2b0094cc016e71555082e3b7801ab2efb7851e3258a44a0739665bfc not found: ID does not exist"
Sep 12 00:58:57.687048 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.687022    2579 scope.go:117] "RemoveContainer" containerID="7ddc3ec7890c8acb6014fe8edf8d1e703a1e42852c3ff094af1f13ac0c367167"
Sep 12 00:58:57.695308 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.695288    2579 scope.go:117] "RemoveContainer" containerID="7ddc3ec7890c8acb6014fe8edf8d1e703a1e42852c3ff094af1f13ac0c367167"
Sep 12 00:58:57.695587 ip-10-0-136-121 kubenswrapper[2579]: E0912 00:58:57.695565    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ddc3ec7890c8acb6014fe8edf8d1e703a1e42852c3ff094af1f13ac0c367167\": container with ID starting with 7ddc3ec7890c8acb6014fe8edf8d1e703a1e42852c3ff094af1f13ac0c367167 not found: ID does not exist" containerID="7ddc3ec7890c8acb6014fe8edf8d1e703a1e42852c3ff094af1f13ac0c367167"
Sep 12 00:58:57.695703 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.695590    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"]
Sep 12 00:58:57.695703 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.695591    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ddc3ec7890c8acb6014fe8edf8d1e703a1e42852c3ff094af1f13ac0c367167"} err="failed to get container status \"7ddc3ec7890c8acb6014fe8edf8d1e703a1e42852c3ff094af1f13ac0c367167\": rpc error: code = NotFound desc = could not find container \"7ddc3ec7890c8acb6014fe8edf8d1e703a1e42852c3ff094af1f13ac0c367167\": container with ID starting with 7ddc3ec7890c8acb6014fe8edf8d1e703a1e42852c3ff094af1f13ac0c367167 not found: ID does not exist"
Sep 12 00:58:57.703170 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.703117    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-64f698-7dd9b56b7-rt8wz"]
Sep 12 00:58:57.714404 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.714380    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"]
Sep 12 00:58:57.719147 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:57.719123    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-64f698-d8ff9b97d-b28nx"]
Sep 12 00:58:59.590559 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:59.590530    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1993dec8-ef1d-4c0b-b1af-2bc5279fe81b" path="/var/lib/kubelet/pods/1993dec8-ef1d-4c0b-b1af-2bc5279fe81b/volumes"
Sep 12 00:58:59.590924 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:58:59.590889    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41700895-1ced-4f45-a3bb-c21318cd30b5" path="/var/lib/kubelet/pods/41700895-1ced-4f45-a3bb-c21318cd30b5/volumes"
Sep 12 00:59:40.760323 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.760290    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"]
Sep 12 00:59:40.760799 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.760662    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73abcfce-3605-4c23-9e39-590c040f4ea3" containerName="revoke-keys"
Sep 12 00:59:40.760799 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.760674    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="73abcfce-3605-4c23-9e39-590c040f4ea3" containerName="revoke-keys"
Sep 12 00:59:40.760799 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.760683    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1993dec8-ef1d-4c0b-b1af-2bc5279fe81b" containerName="payload-processing"
Sep 12 00:59:40.760799 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.760692    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1993dec8-ef1d-4c0b-b1af-2bc5279fe81b" containerName="payload-processing"
Sep 12 00:59:40.760799 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.760711    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41700895-1ced-4f45-a3bb-c21318cd30b5" containerName="payload-pre-processing"
Sep 12 00:59:40.760799 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.760717    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41700895-1ced-4f45-a3bb-c21318cd30b5" containerName="payload-pre-processing"
Sep 12 00:59:40.760799 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.760769    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41700895-1ced-4f45-a3bb-c21318cd30b5" containerName="payload-pre-processing"
Sep 12 00:59:40.760799 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.760778    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1993dec8-ef1d-4c0b-b1af-2bc5279fe81b" containerName="payload-processing"
Sep 12 00:59:40.760799 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.760785    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="73abcfce-3605-4c23-9e39-590c040f4ea3" containerName="revoke-keys"
Sep 12 00:59:40.765472 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.765452    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:40.769381 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.769364    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-8a4fbb-openshift-default-dockercfg-4jqh8\""
Sep 12 00:59:40.780458 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.780436    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"]
Sep 12 00:59:40.916132 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.916099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-workload-certs\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:40.916132 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.916130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-istio-envoy\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:40.916365 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.916157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5093418-eae0-4eaa-bb97-767e71de230e-istiod-ca-cert\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:40.916365 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.916194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-workload-socket\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:40.916365 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.916232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-istio-data\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:40.916365 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.916247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5093418-eae0-4eaa-bb97-767e71de230e-istio-podinfo\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:40.916365 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.916273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwphr\" (UniqueName: \"kubernetes.io/projected/a5093418-eae0-4eaa-bb97-767e71de230e-kube-api-access-vwphr\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:40.916365 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.916293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-credential-socket\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:40.916365 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:40.916311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5093418-eae0-4eaa-bb97-767e71de230e-istio-token\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.017058 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.016980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-credential-socket\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.017058 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.017014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5093418-eae0-4eaa-bb97-767e71de230e-istio-token\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.017266 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.017069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-workload-certs\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.017266 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.017084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-istio-envoy\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.017266 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.017112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5093418-eae0-4eaa-bb97-767e71de230e-istiod-ca-cert\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.017266 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.017133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-workload-socket\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.017266 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.017172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-istio-data\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.017266 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.017191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5093418-eae0-4eaa-bb97-767e71de230e-istio-podinfo\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.017266 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.017210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwphr\" (UniqueName: \"kubernetes.io/projected/a5093418-eae0-4eaa-bb97-767e71de230e-kube-api-access-vwphr\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.017679 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.017481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-workload-certs\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.017679 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.017631    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-istio-data\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.017779 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.017678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-workload-socket\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.017779 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.017759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-credential-socket\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.017941 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.017916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5093418-eae0-4eaa-bb97-767e71de230e-istiod-ca-cert\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.020138 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.020111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-istio-envoy\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.020486 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.020469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5093418-eae0-4eaa-bb97-767e71de230e-istio-podinfo\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.026582 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.026556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5093418-eae0-4eaa-bb97-767e71de230e-istio-token\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.026696 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.026654    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwphr\" (UniqueName: \"kubernetes.io/projected/a5093418-eae0-4eaa-bb97-767e71de230e-kube-api-access-vwphr\") pod \"e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") " pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.076734 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.076707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:41.359459 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.359384    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"]
Sep 12 00:59:41.362530 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:59:41.362505    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5093418_eae0_4eaa_bb97_767e71de230e.slice/crio-6acc121580fc3d583dac666bb269e53bd4f92d5f9e004f87a0603db49101fac3 WatchSource:0}: Error finding container 6acc121580fc3d583dac666bb269e53bd4f92d5f9e004f87a0603db49101fac3: Status 404 returned error can't find the container with id 6acc121580fc3d583dac666bb269e53bd4f92d5f9e004f87a0603db49101fac3
Sep 12 00:59:41.364837 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.364797    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:59:41.364915 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.364863    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:59:41.364915 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.364891    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 00:59:41.891081 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.891044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj" event={"ID":"a5093418-eae0-4eaa-bb97-767e71de230e","Type":"ContainerStarted","Data":"092595da3092db66e2e6d72f85bd70aae095f77d25bc04964af334a79e7f0280"}
Sep 12 00:59:41.891516 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.891096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj" event={"ID":"a5093418-eae0-4eaa-bb97-767e71de230e","Type":"ContainerStarted","Data":"6acc121580fc3d583dac666bb269e53bd4f92d5f9e004f87a0603db49101fac3"}
Sep 12 00:59:41.920491 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:41.920440    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj" podStartSLOduration=1.920423921 podStartE2EDuration="1.920423921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:59:41.917995987 +0000 UTC m=+1006.837287774" watchObservedRunningTime="2026-09-12 00:59:41.920423921 +0000 UTC m=+1006.839715709"
Sep 12 00:59:42.077744 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:42.077706    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:42.083152 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:42.083123    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:42.896409 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:42.896375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:42.897327 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:42.897309    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 00:59:48.683929 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.683897    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"]
Sep 12 00:59:48.687634 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.687589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"
Sep 12 00:59:48.690561 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.690540    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-8a4fbb\""
Sep 12 00:59:48.690689 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.690588    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-8a4fbb-dockercfg-ppzsq\""
Sep 12 00:59:48.696928 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.696907    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"]
Sep 12 00:59:48.700199 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.700179    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"
Sep 12 00:59:48.706013 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.705992    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"]
Sep 12 00:59:48.718202 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.718180    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"]
Sep 12 00:59:48.788754 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.788715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzpn7\" (UniqueName: \"kubernetes.io/projected/65f9c65e-6611-4ca9-a12a-3814225ed0a8-kube-api-access-rzpn7\") pod \"payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf\" (UID: \"65f9c65e-6611-4ca9-a12a-3814225ed0a8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"
Sep 12 00:59:48.789106 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.789081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f9c65e-6611-4ca9-a12a-3814225ed0a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf\" (UID: \"65f9c65e-6611-4ca9-a12a-3814225ed0a8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"
Sep 12 00:59:48.890723 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.890691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gnvt\" (UniqueName: \"kubernetes.io/projected/992472de-ec76-4916-aaa4-c9935b65d25f-kube-api-access-6gnvt\") pod \"payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6\" (UID: \"992472de-ec76-4916-aaa4-c9935b65d25f\") " pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"
Sep 12 00:59:48.890919 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.890792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpn7\" (UniqueName: \"kubernetes.io/projected/65f9c65e-6611-4ca9-a12a-3814225ed0a8-kube-api-access-rzpn7\") pod \"payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf\" (UID: \"65f9c65e-6611-4ca9-a12a-3814225ed0a8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"
Sep 12 00:59:48.890919 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.890827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992472de-ec76-4916-aaa4-c9935b65d25f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6\" (UID: \"992472de-ec76-4916-aaa4-c9935b65d25f\") " pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"
Sep 12 00:59:48.890919 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.890886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f9c65e-6611-4ca9-a12a-3814225ed0a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf\" (UID: \"65f9c65e-6611-4ca9-a12a-3814225ed0a8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"
Sep 12 00:59:48.891554 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.891531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f9c65e-6611-4ca9-a12a-3814225ed0a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf\" (UID: \"65f9c65e-6611-4ca9-a12a-3814225ed0a8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"
Sep 12 00:59:48.903108 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.903078    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpn7\" (UniqueName: \"kubernetes.io/projected/65f9c65e-6611-4ca9-a12a-3814225ed0a8-kube-api-access-rzpn7\") pod \"payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf\" (UID: \"65f9c65e-6611-4ca9-a12a-3814225ed0a8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"
Sep 12 00:59:48.991993 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.991912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992472de-ec76-4916-aaa4-c9935b65d25f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6\" (UID: \"992472de-ec76-4916-aaa4-c9935b65d25f\") " pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"
Sep 12 00:59:48.991993 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.991963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnvt\" (UniqueName: \"kubernetes.io/projected/992472de-ec76-4916-aaa4-c9935b65d25f-kube-api-access-6gnvt\") pod \"payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6\" (UID: \"992472de-ec76-4916-aaa4-c9935b65d25f\") " pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"
Sep 12 00:59:48.992652 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.992596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992472de-ec76-4916-aaa4-c9935b65d25f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6\" (UID: \"992472de-ec76-4916-aaa4-c9935b65d25f\") " pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"
Sep 12 00:59:48.998655 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:48.998630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"
Sep 12 00:59:49.000369 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:49.000344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gnvt\" (UniqueName: \"kubernetes.io/projected/992472de-ec76-4916-aaa4-c9935b65d25f-kube-api-access-6gnvt\") pod \"payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6\" (UID: \"992472de-ec76-4916-aaa4-c9935b65d25f\") " pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"
Sep 12 00:59:49.012079 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:49.012059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"
Sep 12 00:59:49.283308 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:49.283278    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"]
Sep 12 00:59:49.284533 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:59:49.284506    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f9c65e_6611_4ca9_a12a_3814225ed0a8.slice/crio-99d40d97dc2e7e05c1ef51defcf4067ac677f4d340a90d21e2e51eb9872f14e9 WatchSource:0}: Error finding container 99d40d97dc2e7e05c1ef51defcf4067ac677f4d340a90d21e2e51eb9872f14e9: Status 404 returned error can't find the container with id 99d40d97dc2e7e05c1ef51defcf4067ac677f4d340a90d21e2e51eb9872f14e9
Sep 12 00:59:49.301143 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:49.301117    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"]
Sep 12 00:59:49.303143 ip-10-0-136-121 kubenswrapper[2579]: W0912 00:59:49.303117    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992472de_ec76_4916_aaa4_c9935b65d25f.slice/crio-dcc2e801ec792bdc46cbe769565b4d7cdbb40cce99764cfd449e3d4ef9bfc9c3 WatchSource:0}: Error finding container dcc2e801ec792bdc46cbe769565b4d7cdbb40cce99764cfd449e3d4ef9bfc9c3: Status 404 returned error can't find the container with id dcc2e801ec792bdc46cbe769565b4d7cdbb40cce99764cfd449e3d4ef9bfc9c3
Sep 12 00:59:49.934066 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:49.934027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf" event={"ID":"65f9c65e-6611-4ca9-a12a-3814225ed0a8","Type":"ContainerStarted","Data":"b34279ea05d3d2b4fd825c41fe556ccf952478355c2dadebad3c31fe9375e139"}
Sep 12 00:59:49.934521 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:49.934187    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"
Sep 12 00:59:49.934521 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:49.934232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf" event={"ID":"65f9c65e-6611-4ca9-a12a-3814225ed0a8","Type":"ContainerStarted","Data":"99d40d97dc2e7e05c1ef51defcf4067ac677f4d340a90d21e2e51eb9872f14e9"}
Sep 12 00:59:49.935809 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:49.935785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6" event={"ID":"992472de-ec76-4916-aaa4-c9935b65d25f","Type":"ContainerStarted","Data":"ce112c75164cfaeef94974d13359c71f99e7e882bc72276402e26d8825cfdf2c"}
Sep 12 00:59:49.935917 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:49.935829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6" event={"ID":"992472de-ec76-4916-aaa4-c9935b65d25f","Type":"ContainerStarted","Data":"dcc2e801ec792bdc46cbe769565b4d7cdbb40cce99764cfd449e3d4ef9bfc9c3"}
Sep 12 00:59:49.935917 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:49.935892    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"
Sep 12 00:59:49.955388 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:49.955347    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf" podStartSLOduration=1.955335254 podStartE2EDuration="1.955335254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:59:49.953245525 +0000 UTC m=+1014.872537313" watchObservedRunningTime="2026-09-12 00:59:49.955335254 +0000 UTC m=+1014.874627091"
Sep 12 00:59:49.972880 ip-10-0-136-121 kubenswrapper[2579]: I0912 00:59:49.972842    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6" podStartSLOduration=1.97282988 podStartE2EDuration="1.97282988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 00:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 00:59:49.971204003 +0000 UTC m=+1014.890495791" watchObservedRunningTime="2026-09-12 00:59:49.97282988 +0000 UTC m=+1014.892121666"
Sep 12 01:00:00.151749 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.151704    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819580-vmc6q"]
Sep 12 01:00:00.156647 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.156594    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-8a4fbb-29819580-wkp4d"]
Sep 12 01:00:00.156769 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.156707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-vmc6q"
Sep 12 01:00:00.159518 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.159496    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2rn9k\""
Sep 12 01:00:00.160145 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.160126    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-8a4fbb-29819580-wkp4d"
Sep 12 01:00:00.166082 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.166060    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819580-vmc6q"]
Sep 12 01:00:00.168436 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.168407    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-8a4fbb-29819580-wkp4d"]
Sep 12 01:00:00.170831 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.170813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-vmc6q"
Sep 12 01:00:00.176064 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.176043    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bf547a-29819580-zflwx"]
Sep 12 01:00:00.178886 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.178866    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-8a4fbb-29819580-wkp4d"
Sep 12 01:00:00.179871 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.179852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bf547a-29819580-zflwx"
Sep 12 01:00:00.189037 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.189016    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bf547a-29819580-zflwx"]
Sep 12 01:00:00.228005 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.227953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bf547a-29819580-zflwx"
Sep 12 01:00:00.457587 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.457542    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819580-vmc6q"]
Sep 12 01:00:00.460080 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:00:00.460029    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ff60c3_c329_4fa2_bbc8_f624a02dafe1.slice/crio-41bfddca89870b1dbed5ea3f2539e4a40b7cbcada3bf02cd6cd29b30766cf187 WatchSource:0}: Error finding container 41bfddca89870b1dbed5ea3f2539e4a40b7cbcada3bf02cd6cd29b30766cf187: Status 404 returned error can't find the container with id 41bfddca89870b1dbed5ea3f2539e4a40b7cbcada3bf02cd6cd29b30766cf187
Sep 12 01:00:00.485180 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.485161    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-8a4fbb-29819580-wkp4d"]
Sep 12 01:00:00.493659 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:00:00.493636    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f20ef8_0c32_41ff_8eb1_fd95a9628342.slice/crio-20d19f21bd59ff4fcf6556090f30b1b1c374292a94aea8a0c5e9d7876cbc3aad WatchSource:0}: Error finding container 20d19f21bd59ff4fcf6556090f30b1b1c374292a94aea8a0c5e9d7876cbc3aad: Status 404 returned error can't find the container with id 20d19f21bd59ff4fcf6556090f30b1b1c374292a94aea8a0c5e9d7876cbc3aad
Sep 12 01:00:00.523138 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.519861    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bf547a-29819580-zflwx"]
Sep 12 01:00:00.943062 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.943039    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"
Sep 12 01:00:00.943200 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.943101    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"
Sep 12 01:00:00.991646 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.991548    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-vmc6q" event={"ID":"b1ff60c3-c329-4fa2-bbc8-f624a02dafe1","Type":"ContainerStarted","Data":"0c4ebf6c592d758a96a3ee8f47969d1dff844ab807f9925340c0cf2c9d38d0b2"}
Sep 12 01:00:00.991874 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.991844    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-vmc6q" event={"ID":"b1ff60c3-c329-4fa2-bbc8-f624a02dafe1","Type":"ContainerStarted","Data":"41bfddca89870b1dbed5ea3f2539e4a40b7cbcada3bf02cd6cd29b30766cf187"}
Sep 12 01:00:00.993556 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.993535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bf547a-29819580-zflwx" event={"ID":"c7d4dea0-5119-4846-81e7-7603120d4962","Type":"ContainerStarted","Data":"eacc3e264164b5dfb54e6dbfb7f43eb53886c07d3220a5f85ac4042510f74612"}
Sep 12 01:00:00.993670 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.993579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bf547a-29819580-zflwx" event={"ID":"c7d4dea0-5119-4846-81e7-7603120d4962","Type":"ContainerStarted","Data":"a8df852ab30514d464ac44000d8b7a446d566f792d3abf8887a6d3d4964e7163"}
Sep 12 01:00:00.995319 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.995302    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-8a4fbb-29819580-wkp4d" event={"ID":"03f20ef8-0c32-41ff-8eb1-fd95a9628342","Type":"ContainerStarted","Data":"4ef57962186e503801a651e5c1a338d53d12a2acb65cb9fb8ec01317cf2dc0bb"}
Sep 12 01:00:00.995429 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:00.995418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-8a4fbb-29819580-wkp4d" event={"ID":"03f20ef8-0c32-41ff-8eb1-fd95a9628342","Type":"ContainerStarted","Data":"20d19f21bd59ff4fcf6556090f30b1b1c374292a94aea8a0c5e9d7876cbc3aad"}
Sep 12 01:00:01.009705 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:01.009202    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-vmc6q" podStartSLOduration=1.009185038 podStartE2EDuration="1.009185038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:00:01.007983886 +0000 UTC m=+1025.927275675" watchObservedRunningTime="2026-09-12 01:00:01.009185038 +0000 UTC m=+1025.928476826"
Sep 12 01:00:01.028284 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:01.028222    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bf547a-29819580-zflwx" podStartSLOduration=1.028201113 podStartE2EDuration="1.028201113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:00:01.025692016 +0000 UTC m=+1025.944983818" watchObservedRunningTime="2026-09-12 01:00:01.028201113 +0000 UTC m=+1025.947492902"
Sep 12 01:00:01.042038 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:01.041989    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-8a4fbb-29819580-wkp4d" podStartSLOduration=1.041972994 podStartE2EDuration="1.041972994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:00:01.041164847 +0000 UTC m=+1025.960456656" watchObservedRunningTime="2026-09-12 01:00:01.041972994 +0000 UTC m=+1025.961264781"
Sep 12 01:00:02.019426 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:02.018964    2579 generic.go:356] "Generic (PLEG): container finished" podID="03f20ef8-0c32-41ff-8eb1-fd95a9628342" containerID="4ef57962186e503801a651e5c1a338d53d12a2acb65cb9fb8ec01317cf2dc0bb" exitCode=0
Sep 12 01:00:02.019426 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:02.019075    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-8a4fbb-29819580-wkp4d" event={"ID":"03f20ef8-0c32-41ff-8eb1-fd95a9628342","Type":"ContainerDied","Data":"4ef57962186e503801a651e5c1a338d53d12a2acb65cb9fb8ec01317cf2dc0bb"}
Sep 12 01:00:02.024228 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:02.023997    2579 generic.go:356] "Generic (PLEG): container finished" podID="b1ff60c3-c329-4fa2-bbc8-f624a02dafe1" containerID="0c4ebf6c592d758a96a3ee8f47969d1dff844ab807f9925340c0cf2c9d38d0b2" exitCode=0
Sep 12 01:00:02.024228 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:02.024097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-vmc6q" event={"ID":"b1ff60c3-c329-4fa2-bbc8-f624a02dafe1","Type":"ContainerDied","Data":"0c4ebf6c592d758a96a3ee8f47969d1dff844ab807f9925340c0cf2c9d38d0b2"}
Sep 12 01:00:02.026452 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:02.026292    2579 generic.go:356] "Generic (PLEG): container finished" podID="c7d4dea0-5119-4846-81e7-7603120d4962" containerID="eacc3e264164b5dfb54e6dbfb7f43eb53886c07d3220a5f85ac4042510f74612" exitCode=0
Sep 12 01:00:02.026452 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:02.026334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bf547a-29819580-zflwx" event={"ID":"c7d4dea0-5119-4846-81e7-7603120d4962","Type":"ContainerDied","Data":"eacc3e264164b5dfb54e6dbfb7f43eb53886c07d3220a5f85ac4042510f74612"}
Sep 12 01:00:03.222625 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:03.216785    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bf547a-29819580-zflwx"
Sep 12 01:00:03.270713 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:03.270691    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-vmc6q"
Sep 12 01:00:03.273421 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:03.273400    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-8a4fbb-29819580-wkp4d"
Sep 12 01:00:04.037560 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:04.037485    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-vmc6q"
Sep 12 01:00:04.037560 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:04.037519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819580-vmc6q" event={"ID":"b1ff60c3-c329-4fa2-bbc8-f624a02dafe1","Type":"ContainerDied","Data":"41bfddca89870b1dbed5ea3f2539e4a40b7cbcada3bf02cd6cd29b30766cf187"}
Sep 12 01:00:04.037845 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:04.037573    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41bfddca89870b1dbed5ea3f2539e4a40b7cbcada3bf02cd6cd29b30766cf187"
Sep 12 01:00:04.038696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:04.038680    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bf547a-29819580-zflwx"
Sep 12 01:00:04.038891 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:04.038764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bf547a-29819580-zflwx" event={"ID":"c7d4dea0-5119-4846-81e7-7603120d4962","Type":"ContainerDied","Data":"a8df852ab30514d464ac44000d8b7a446d566f792d3abf8887a6d3d4964e7163"}
Sep 12 01:00:04.038972 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:04.038915    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8df852ab30514d464ac44000d8b7a446d566f792d3abf8887a6d3d4964e7163"
Sep 12 01:00:04.040385 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:04.040328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-8a4fbb-29819580-wkp4d" event={"ID":"03f20ef8-0c32-41ff-8eb1-fd95a9628342","Type":"ContainerDied","Data":"20d19f21bd59ff4fcf6556090f30b1b1c374292a94aea8a0c5e9d7876cbc3aad"}
Sep 12 01:00:04.040385 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:04.040348    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-8a4fbb-29819580-wkp4d"
Sep 12 01:00:04.040385 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:04.040365    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20d19f21bd59ff4fcf6556090f30b1b1c374292a94aea8a0c5e9d7876cbc3aad"
Sep 12 01:00:09.067785 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:09.067755    2579 generic.go:356] "Generic (PLEG): container finished" podID="a5093418-eae0-4eaa-bb97-767e71de230e" containerID="092595da3092db66e2e6d72f85bd70aae095f77d25bc04964af334a79e7f0280" exitCode=0
Sep 12 01:00:09.068179 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:09.067828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj" event={"ID":"a5093418-eae0-4eaa-bb97-767e71de230e","Type":"ContainerDied","Data":"092595da3092db66e2e6d72f85bd70aae095f77d25bc04964af334a79e7f0280"}
Sep 12 01:00:09.068223 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:09.068183    2579 scope.go:117] "RemoveContainer" containerID="092595da3092db66e2e6d72f85bd70aae095f77d25bc04964af334a79e7f0280"
Sep 12 01:00:09.069269 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:09.069242    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 01:00:09.069366 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:09.069326    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 01:00:09.069429 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:09.069373    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 01:00:10.075231 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:10.075199    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj" event={"ID":"a5093418-eae0-4eaa-bb97-767e71de230e","Type":"ContainerStarted","Data":"b110bd7a6ac48479173a5fb04038e820656fe39853e8170609a87e01acb62062"}
Sep 12 01:00:10.077656 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:10.077629    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 01:00:10.078656 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:10.078635    2579 patch_prober.go:28] interesting pod/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 12 01:00:10.078715 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:10.078698    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj" podUID="a5093418-eae0-4eaa-bb97-767e71de230e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 12 01:00:11.077449 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:11.077416    2579 patch_prober.go:28] interesting pod/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 12 01:00:11.077859 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:11.077475    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj" podUID="a5093418-eae0-4eaa-bb97-767e71de230e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 12 01:00:12.077489 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:12.077454    2579 patch_prober.go:28] interesting pod/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 12 01:00:12.077892 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:12.077510    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj" podUID="a5093418-eae0-4eaa-bb97-767e71de230e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 12 01:00:12.896782 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:12.896752    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 01:00:13.077834 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:13.077795    2579 patch_prober.go:28] interesting pod/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 12 01:00:13.078294 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:13.077854    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj" podUID="a5093418-eae0-4eaa-bb97-767e71de230e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 12 01:00:14.077285 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:14.077246    2579 patch_prober.go:28] interesting pod/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 12 01:00:14.077460 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:14.077318    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj" podUID="a5093418-eae0-4eaa-bb97-767e71de230e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 12 01:00:15.202481 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:15.202447    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 01:00:15.204531 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:15.204505    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 01:00:33.011649 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.011534    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"]
Sep 12 01:00:33.012039 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.011991    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d4dea0-5119-4846-81e7-7603120d4962" containerName="cleanup"
Sep 12 01:00:33.012039 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.012006    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d4dea0-5119-4846-81e7-7603120d4962" containerName="cleanup"
Sep 12 01:00:33.012039 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.012020    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f20ef8-0c32-41ff-8eb1-fd95a9628342" containerName="cleanup"
Sep 12 01:00:33.012039 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.012025    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f20ef8-0c32-41ff-8eb1-fd95a9628342" containerName="cleanup"
Sep 12 01:00:33.012039 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.012037    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ff60c3-c329-4fa2-bbc8-f624a02dafe1" containerName="cleanup"
Sep 12 01:00:33.012039 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.012042    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ff60c3-c329-4fa2-bbc8-f624a02dafe1" containerName="cleanup"
Sep 12 01:00:33.012233 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.012101    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f20ef8-0c32-41ff-8eb1-fd95a9628342" containerName="cleanup"
Sep 12 01:00:33.012233 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.012109    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d4dea0-5119-4846-81e7-7603120d4962" containerName="cleanup"
Sep 12 01:00:33.012233 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.012114    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ff60c3-c329-4fa2-bbc8-f624a02dafe1" containerName="cleanup"
Sep 12 01:00:33.015304 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.015287    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"
Sep 12 01:00:33.017986 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.017964    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9eca3408\""
Sep 12 01:00:33.018163 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.018142    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9eca3408-dockercfg-cxznx\""
Sep 12 01:00:33.029296 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.029276    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"]
Sep 12 01:00:33.104676 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.104650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krjmk\" (UniqueName: \"kubernetes.io/projected/e112c39f-2ec5-47a3-a32d-1822cbbce621-kube-api-access-krjmk\") pod \"payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w\" (UID: \"e112c39f-2ec5-47a3-a32d-1822cbbce621\") " pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"
Sep 12 01:00:33.104821 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.104700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e112c39f-2ec5-47a3-a32d-1822cbbce621-plugins-config-volume\") pod \"payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w\" (UID: \"e112c39f-2ec5-47a3-a32d-1822cbbce621\") " pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"
Sep 12 01:00:33.205711 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.205675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e112c39f-2ec5-47a3-a32d-1822cbbce621-plugins-config-volume\") pod \"payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w\" (UID: \"e112c39f-2ec5-47a3-a32d-1822cbbce621\") " pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"
Sep 12 01:00:33.205894 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.205797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krjmk\" (UniqueName: \"kubernetes.io/projected/e112c39f-2ec5-47a3-a32d-1822cbbce621-kube-api-access-krjmk\") pod \"payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w\" (UID: \"e112c39f-2ec5-47a3-a32d-1822cbbce621\") " pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"
Sep 12 01:00:33.206586 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.206540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e112c39f-2ec5-47a3-a32d-1822cbbce621-plugins-config-volume\") pod \"payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w\" (UID: \"e112c39f-2ec5-47a3-a32d-1822cbbce621\") " pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"
Sep 12 01:00:33.219207 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.219118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krjmk\" (UniqueName: \"kubernetes.io/projected/e112c39f-2ec5-47a3-a32d-1822cbbce621-kube-api-access-krjmk\") pod \"payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w\" (UID: \"e112c39f-2ec5-47a3-a32d-1822cbbce621\") " pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"
Sep 12 01:00:33.325649 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.325537    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"
Sep 12 01:00:33.597616 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:33.597572    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"]
Sep 12 01:00:33.599541 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:00:33.599515    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode112c39f_2ec5_47a3_a32d_1822cbbce621.slice/crio-360a7e790cc14efd653ebaf6989a751d688541557dd41cf66762f2e074fcbd88 WatchSource:0}: Error finding container 360a7e790cc14efd653ebaf6989a751d688541557dd41cf66762f2e074fcbd88: Status 404 returned error can't find the container with id 360a7e790cc14efd653ebaf6989a751d688541557dd41cf66762f2e074fcbd88
Sep 12 01:00:34.207973 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:34.207924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w" event={"ID":"e112c39f-2ec5-47a3-a32d-1822cbbce621","Type":"ContainerStarted","Data":"ac63721e2d048d6e9bc7d8de1d7ccb327af1dd793df15f3a6b806a7206f19781"}
Sep 12 01:00:34.208444 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:34.207998    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"
Sep 12 01:00:34.208444 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:34.208024    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w" event={"ID":"e112c39f-2ec5-47a3-a32d-1822cbbce621","Type":"ContainerStarted","Data":"360a7e790cc14efd653ebaf6989a751d688541557dd41cf66762f2e074fcbd88"}
Sep 12 01:00:34.224740 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:34.224682    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w" podStartSLOduration=2.22466369 podStartE2EDuration="2.22466369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:00:34.223864805 +0000 UTC m=+1059.143156583" watchObservedRunningTime="2026-09-12 01:00:34.22466369 +0000 UTC m=+1059.143955480"
Sep 12 01:00:45.215083 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:00:45.215053    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"
Sep 12 01:01:15.654344 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:15.654131    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"]
Sep 12 01:01:15.654959 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:15.654445    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w" podUID="e112c39f-2ec5-47a3-a32d-1822cbbce621" containerName="payload-processing" containerID="cri-o://ac63721e2d048d6e9bc7d8de1d7ccb327af1dd793df15f3a6b806a7206f19781" gracePeriod=30
Sep 12 01:01:15.920734 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:15.920703    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"
Sep 12 01:01:16.081845 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:16.081812    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e112c39f-2ec5-47a3-a32d-1822cbbce621-plugins-config-volume\") pod \"e112c39f-2ec5-47a3-a32d-1822cbbce621\" (UID: \"e112c39f-2ec5-47a3-a32d-1822cbbce621\") "
Sep 12 01:01:16.082022 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:16.081856    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krjmk\" (UniqueName: \"kubernetes.io/projected/e112c39f-2ec5-47a3-a32d-1822cbbce621-kube-api-access-krjmk\") pod \"e112c39f-2ec5-47a3-a32d-1822cbbce621\" (UID: \"e112c39f-2ec5-47a3-a32d-1822cbbce621\") "
Sep 12 01:01:16.082183 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:16.082158    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e112c39f-2ec5-47a3-a32d-1822cbbce621-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e112c39f-2ec5-47a3-a32d-1822cbbce621" (UID: "e112c39f-2ec5-47a3-a32d-1822cbbce621"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:01:16.083890 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:16.083868    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e112c39f-2ec5-47a3-a32d-1822cbbce621-kube-api-access-krjmk" (OuterVolumeSpecName: "kube-api-access-krjmk") pod "e112c39f-2ec5-47a3-a32d-1822cbbce621" (UID: "e112c39f-2ec5-47a3-a32d-1822cbbce621"). InnerVolumeSpecName "kube-api-access-krjmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:01:16.182569 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:16.182542    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e112c39f-2ec5-47a3-a32d-1822cbbce621-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:16.182569 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:16.182566    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krjmk\" (UniqueName: \"kubernetes.io/projected/e112c39f-2ec5-47a3-a32d-1822cbbce621-kube-api-access-krjmk\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:16.434401 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:16.434323    2579 generic.go:356] "Generic (PLEG): container finished" podID="e112c39f-2ec5-47a3-a32d-1822cbbce621" containerID="ac63721e2d048d6e9bc7d8de1d7ccb327af1dd793df15f3a6b806a7206f19781" exitCode=0
Sep 12 01:01:16.434401 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:16.434365    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"
Sep 12 01:01:16.434626 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:16.434401    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w" event={"ID":"e112c39f-2ec5-47a3-a32d-1822cbbce621","Type":"ContainerDied","Data":"ac63721e2d048d6e9bc7d8de1d7ccb327af1dd793df15f3a6b806a7206f19781"}
Sep 12 01:01:16.434626 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:16.434459    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w" event={"ID":"e112c39f-2ec5-47a3-a32d-1822cbbce621","Type":"ContainerDied","Data":"360a7e790cc14efd653ebaf6989a751d688541557dd41cf66762f2e074fcbd88"}
Sep 12 01:01:16.434626 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:16.434491    2579 scope.go:117] "RemoveContainer" containerID="ac63721e2d048d6e9bc7d8de1d7ccb327af1dd793df15f3a6b806a7206f19781"
Sep 12 01:01:16.444116 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:16.444093    2579 scope.go:117] "RemoveContainer" containerID="ac63721e2d048d6e9bc7d8de1d7ccb327af1dd793df15f3a6b806a7206f19781"
Sep 12 01:01:16.444369 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:01:16.444353    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac63721e2d048d6e9bc7d8de1d7ccb327af1dd793df15f3a6b806a7206f19781\": container with ID starting with ac63721e2d048d6e9bc7d8de1d7ccb327af1dd793df15f3a6b806a7206f19781 not found: ID does not exist" containerID="ac63721e2d048d6e9bc7d8de1d7ccb327af1dd793df15f3a6b806a7206f19781"
Sep 12 01:01:16.444419 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:16.444376    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac63721e2d048d6e9bc7d8de1d7ccb327af1dd793df15f3a6b806a7206f19781"} err="failed to get container status \"ac63721e2d048d6e9bc7d8de1d7ccb327af1dd793df15f3a6b806a7206f19781\": rpc error: code = NotFound desc = could not find container \"ac63721e2d048d6e9bc7d8de1d7ccb327af1dd793df15f3a6b806a7206f19781\": container with ID starting with ac63721e2d048d6e9bc7d8de1d7ccb327af1dd793df15f3a6b806a7206f19781 not found: ID does not exist"
Sep 12 01:01:16.457426 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:16.457405    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"]
Sep 12 01:01:16.463402 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:16.463380    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9eca3408-5d5876c4d5-9tb4w"]
Sep 12 01:01:17.592115 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:17.592084    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e112c39f-2ec5-47a3-a32d-1822cbbce621" path="/var/lib/kubelet/pods/e112c39f-2ec5-47a3-a32d-1822cbbce621/volumes"
Sep 12 01:01:42.531905 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.531870    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-8a4fbb-29819580-wkp4d"]
Sep 12 01:01:42.538308 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.538282    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-8a4fbb-29819580-wkp4d"]
Sep 12 01:01:42.564545 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.564505    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"]
Sep 12 01:01:42.564826 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.564775    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6" podUID="992472de-ec76-4916-aaa4-c9935b65d25f" containerName="payload-processing" containerID="cri-o://ce112c75164cfaeef94974d13359c71f99e7e882bc72276402e26d8825cfdf2c" gracePeriod=30
Sep 12 01:01:42.573656 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.573377    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"]
Sep 12 01:01:42.573982 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.573824    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf" podUID="65f9c65e-6611-4ca9-a12a-3814225ed0a8" containerName="payload-pre-processing" containerID="cri-o://b34279ea05d3d2b4fd825c41fe556ccf952478355c2dadebad3c31fe9375e139" gracePeriod=30
Sep 12 01:01:42.839082 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.839059    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"
Sep 12 01:01:42.858500 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.858477    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"
Sep 12 01:01:42.928623 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.928501    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzpn7\" (UniqueName: \"kubernetes.io/projected/65f9c65e-6611-4ca9-a12a-3814225ed0a8-kube-api-access-rzpn7\") pod \"65f9c65e-6611-4ca9-a12a-3814225ed0a8\" (UID: \"65f9c65e-6611-4ca9-a12a-3814225ed0a8\") "
Sep 12 01:01:42.928623 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.928579    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992472de-ec76-4916-aaa4-c9935b65d25f-plugins-config-volume\") pod \"992472de-ec76-4916-aaa4-c9935b65d25f\" (UID: \"992472de-ec76-4916-aaa4-c9935b65d25f\") "
Sep 12 01:01:42.928931 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.928636    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gnvt\" (UniqueName: \"kubernetes.io/projected/992472de-ec76-4916-aaa4-c9935b65d25f-kube-api-access-6gnvt\") pod \"992472de-ec76-4916-aaa4-c9935b65d25f\" (UID: \"992472de-ec76-4916-aaa4-c9935b65d25f\") "
Sep 12 01:01:42.928931 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.928692    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f9c65e-6611-4ca9-a12a-3814225ed0a8-plugins-config-volume\") pod \"65f9c65e-6611-4ca9-a12a-3814225ed0a8\" (UID: \"65f9c65e-6611-4ca9-a12a-3814225ed0a8\") "
Sep 12 01:01:42.933624 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.929279    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f9c65e-6611-4ca9-a12a-3814225ed0a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65f9c65e-6611-4ca9-a12a-3814225ed0a8" (UID: "65f9c65e-6611-4ca9-a12a-3814225ed0a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:01:42.933624 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.929833    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992472de-ec76-4916-aaa4-c9935b65d25f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "992472de-ec76-4916-aaa4-c9935b65d25f" (UID: "992472de-ec76-4916-aaa4-c9935b65d25f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:01:42.933624 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.932956    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f9c65e-6611-4ca9-a12a-3814225ed0a8-kube-api-access-rzpn7" (OuterVolumeSpecName: "kube-api-access-rzpn7") pod "65f9c65e-6611-4ca9-a12a-3814225ed0a8" (UID: "65f9c65e-6611-4ca9-a12a-3814225ed0a8"). InnerVolumeSpecName "kube-api-access-rzpn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:01:42.933624 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:42.933069    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992472de-ec76-4916-aaa4-c9935b65d25f-kube-api-access-6gnvt" (OuterVolumeSpecName: "kube-api-access-6gnvt") pod "992472de-ec76-4916-aaa4-c9935b65d25f" (UID: "992472de-ec76-4916-aaa4-c9935b65d25f"). InnerVolumeSpecName "kube-api-access-6gnvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:01:43.030022 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.029985    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gnvt\" (UniqueName: \"kubernetes.io/projected/992472de-ec76-4916-aaa4-c9935b65d25f-kube-api-access-6gnvt\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:43.030022 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.030017    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f9c65e-6611-4ca9-a12a-3814225ed0a8-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:43.030022 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.030026    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzpn7\" (UniqueName: \"kubernetes.io/projected/65f9c65e-6611-4ca9-a12a-3814225ed0a8-kube-api-access-rzpn7\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:43.030272 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.030035    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992472de-ec76-4916-aaa4-c9935b65d25f-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:43.590904 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.590877    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03f20ef8-0c32-41ff-8eb1-fd95a9628342" path="/var/lib/kubelet/pods/03f20ef8-0c32-41ff-8eb1-fd95a9628342/volumes"
Sep 12 01:01:43.591709 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.591685    2579 generic.go:356] "Generic (PLEG): container finished" podID="992472de-ec76-4916-aaa4-c9935b65d25f" containerID="ce112c75164cfaeef94974d13359c71f99e7e882bc72276402e26d8825cfdf2c" exitCode=0
Sep 12 01:01:43.591831 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.591719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6" event={"ID":"992472de-ec76-4916-aaa4-c9935b65d25f","Type":"ContainerDied","Data":"ce112c75164cfaeef94974d13359c71f99e7e882bc72276402e26d8825cfdf2c"}
Sep 12 01:01:43.591831 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.591738    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"
Sep 12 01:01:43.591831 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.591781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6" event={"ID":"992472de-ec76-4916-aaa4-c9935b65d25f","Type":"ContainerDied","Data":"dcc2e801ec792bdc46cbe769565b4d7cdbb40cce99764cfd449e3d4ef9bfc9c3"}
Sep 12 01:01:43.591831 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.591813    2579 scope.go:117] "RemoveContainer" containerID="ce112c75164cfaeef94974d13359c71f99e7e882bc72276402e26d8825cfdf2c"
Sep 12 01:01:43.593261 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.593236    2579 generic.go:356] "Generic (PLEG): container finished" podID="65f9c65e-6611-4ca9-a12a-3814225ed0a8" containerID="b34279ea05d3d2b4fd825c41fe556ccf952478355c2dadebad3c31fe9375e139" exitCode=0
Sep 12 01:01:43.593333 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.593289    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"
Sep 12 01:01:43.593333 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.593301    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf" event={"ID":"65f9c65e-6611-4ca9-a12a-3814225ed0a8","Type":"ContainerDied","Data":"b34279ea05d3d2b4fd825c41fe556ccf952478355c2dadebad3c31fe9375e139"}
Sep 12 01:01:43.593436 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.593372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf" event={"ID":"65f9c65e-6611-4ca9-a12a-3814225ed0a8","Type":"ContainerDied","Data":"99d40d97dc2e7e05c1ef51defcf4067ac677f4d340a90d21e2e51eb9872f14e9"}
Sep 12 01:01:43.615853 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.615829    2579 scope.go:117] "RemoveContainer" containerID="ce112c75164cfaeef94974d13359c71f99e7e882bc72276402e26d8825cfdf2c"
Sep 12 01:01:43.616250 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:01:43.616227    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce112c75164cfaeef94974d13359c71f99e7e882bc72276402e26d8825cfdf2c\": container with ID starting with ce112c75164cfaeef94974d13359c71f99e7e882bc72276402e26d8825cfdf2c not found: ID does not exist" containerID="ce112c75164cfaeef94974d13359c71f99e7e882bc72276402e26d8825cfdf2c"
Sep 12 01:01:43.616349 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.616255    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce112c75164cfaeef94974d13359c71f99e7e882bc72276402e26d8825cfdf2c"} err="failed to get container status \"ce112c75164cfaeef94974d13359c71f99e7e882bc72276402e26d8825cfdf2c\": rpc error: code = NotFound desc = could not find container \"ce112c75164cfaeef94974d13359c71f99e7e882bc72276402e26d8825cfdf2c\": container with ID starting with ce112c75164cfaeef94974d13359c71f99e7e882bc72276402e26d8825cfdf2c not found: ID does not exist"
Sep 12 01:01:43.616349 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.616271    2579 scope.go:117] "RemoveContainer" containerID="b34279ea05d3d2b4fd825c41fe556ccf952478355c2dadebad3c31fe9375e139"
Sep 12 01:01:43.624083 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.624063    2579 scope.go:117] "RemoveContainer" containerID="b34279ea05d3d2b4fd825c41fe556ccf952478355c2dadebad3c31fe9375e139"
Sep 12 01:01:43.624373 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:01:43.624353    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34279ea05d3d2b4fd825c41fe556ccf952478355c2dadebad3c31fe9375e139\": container with ID starting with b34279ea05d3d2b4fd825c41fe556ccf952478355c2dadebad3c31fe9375e139 not found: ID does not exist" containerID="b34279ea05d3d2b4fd825c41fe556ccf952478355c2dadebad3c31fe9375e139"
Sep 12 01:01:43.624438 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.624378    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34279ea05d3d2b4fd825c41fe556ccf952478355c2dadebad3c31fe9375e139"} err="failed to get container status \"b34279ea05d3d2b4fd825c41fe556ccf952478355c2dadebad3c31fe9375e139\": rpc error: code = NotFound desc = could not find container \"b34279ea05d3d2b4fd825c41fe556ccf952478355c2dadebad3c31fe9375e139\": container with ID starting with b34279ea05d3d2b4fd825c41fe556ccf952478355c2dadebad3c31fe9375e139 not found: ID does not exist"
Sep 12 01:01:43.631708 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.631681    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"]
Sep 12 01:01:43.638462 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.638442    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8a4fbb-86d587497-k5rt6"]
Sep 12 01:01:43.648321 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.648301    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"]
Sep 12 01:01:43.652307 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:43.652287    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-8a4fbb-6dc57499d-pzrgf"]
Sep 12 01:01:44.616779 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.616750    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"]
Sep 12 01:01:44.617250 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.617169    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f9c65e-6611-4ca9-a12a-3814225ed0a8" containerName="payload-pre-processing"
Sep 12 01:01:44.617250 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.617183    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f9c65e-6611-4ca9-a12a-3814225ed0a8" containerName="payload-pre-processing"
Sep 12 01:01:44.617250 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.617190    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="992472de-ec76-4916-aaa4-c9935b65d25f" containerName="payload-processing"
Sep 12 01:01:44.617250 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.617198    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="992472de-ec76-4916-aaa4-c9935b65d25f" containerName="payload-processing"
Sep 12 01:01:44.617250 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.617212    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e112c39f-2ec5-47a3-a32d-1822cbbce621" containerName="payload-processing"
Sep 12 01:01:44.617250 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.617218    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e112c39f-2ec5-47a3-a32d-1822cbbce621" containerName="payload-processing"
Sep 12 01:01:44.617555 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.617273    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e112c39f-2ec5-47a3-a32d-1822cbbce621" containerName="payload-processing"
Sep 12 01:01:44.617555 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.617282    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f9c65e-6611-4ca9-a12a-3814225ed0a8" containerName="payload-pre-processing"
Sep 12 01:01:44.617555 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.617289    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="992472de-ec76-4916-aaa4-c9935b65d25f" containerName="payload-processing"
Sep 12 01:01:44.621579 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.621560    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"]
Sep 12 01:01:44.621722 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.621691    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"
Sep 12 01:01:44.624213 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.624194    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-8e62c2\""
Sep 12 01:01:44.624303 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.624210    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-8e62c2-dockercfg-vx25d\""
Sep 12 01:01:44.624647 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.624630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"
Sep 12 01:01:44.633457 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.633427    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"]
Sep 12 01:01:44.635382 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.635358    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"]
Sep 12 01:01:44.746079 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.746038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbcdc926-82c2-41d7-a231-81280d5d7864-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq\" (UID: \"dbcdc926-82c2-41d7-a231-81280d5d7864\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"
Sep 12 01:01:44.746280 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.746171    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stlht\" (UniqueName: \"kubernetes.io/projected/e9aeea73-045e-4e21-b0ce-644cf60649ad-kube-api-access-stlht\") pod \"payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879\" (UID: \"e9aeea73-045e-4e21-b0ce-644cf60649ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"
Sep 12 01:01:44.746346 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.746284    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9aeea73-045e-4e21-b0ce-644cf60649ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879\" (UID: \"e9aeea73-045e-4e21-b0ce-644cf60649ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"
Sep 12 01:01:44.746346 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.746308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4l4c\" (UniqueName: \"kubernetes.io/projected/dbcdc926-82c2-41d7-a231-81280d5d7864-kube-api-access-q4l4c\") pod \"payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq\" (UID: \"dbcdc926-82c2-41d7-a231-81280d5d7864\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"
Sep 12 01:01:44.847539 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.847504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stlht\" (UniqueName: \"kubernetes.io/projected/e9aeea73-045e-4e21-b0ce-644cf60649ad-kube-api-access-stlht\") pod \"payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879\" (UID: \"e9aeea73-045e-4e21-b0ce-644cf60649ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"
Sep 12 01:01:44.847727 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.847575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9aeea73-045e-4e21-b0ce-644cf60649ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879\" (UID: \"e9aeea73-045e-4e21-b0ce-644cf60649ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"
Sep 12 01:01:44.847727 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.847593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4l4c\" (UniqueName: \"kubernetes.io/projected/dbcdc926-82c2-41d7-a231-81280d5d7864-kube-api-access-q4l4c\") pod \"payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq\" (UID: \"dbcdc926-82c2-41d7-a231-81280d5d7864\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"
Sep 12 01:01:44.847727 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.847647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbcdc926-82c2-41d7-a231-81280d5d7864-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq\" (UID: \"dbcdc926-82c2-41d7-a231-81280d5d7864\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"
Sep 12 01:01:44.848318 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.848292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9aeea73-045e-4e21-b0ce-644cf60649ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879\" (UID: \"e9aeea73-045e-4e21-b0ce-644cf60649ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"
Sep 12 01:01:44.848318 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.848303    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbcdc926-82c2-41d7-a231-81280d5d7864-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq\" (UID: \"dbcdc926-82c2-41d7-a231-81280d5d7864\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"
Sep 12 01:01:44.855703 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.855682    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4l4c\" (UniqueName: \"kubernetes.io/projected/dbcdc926-82c2-41d7-a231-81280d5d7864-kube-api-access-q4l4c\") pod \"payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq\" (UID: \"dbcdc926-82c2-41d7-a231-81280d5d7864\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"
Sep 12 01:01:44.855787 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.855770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stlht\" (UniqueName: \"kubernetes.io/projected/e9aeea73-045e-4e21-b0ce-644cf60649ad-kube-api-access-stlht\") pod \"payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879\" (UID: \"e9aeea73-045e-4e21-b0ce-644cf60649ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"
Sep 12 01:01:44.933857 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.933831    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"
Sep 12 01:01:44.941709 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:44.941689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"
Sep 12 01:01:45.219325 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:45.219226    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"]
Sep 12 01:01:45.221867 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:01:45.221838    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcdc926_82c2_41d7_a231_81280d5d7864.slice/crio-00cdc57e51942f415f59ce9ba49149de894d5a8b0455a6e96696664b0c6f4bd7 WatchSource:0}: Error finding container 00cdc57e51942f415f59ce9ba49149de894d5a8b0455a6e96696664b0c6f4bd7: Status 404 returned error can't find the container with id 00cdc57e51942f415f59ce9ba49149de894d5a8b0455a6e96696664b0c6f4bd7
Sep 12 01:01:45.227285 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:45.227257    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"]
Sep 12 01:01:45.228685 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:01:45.228661    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9aeea73_045e_4e21_b0ce_644cf60649ad.slice/crio-395f063817b2601473a4c33a6a357311bafefa3a918ec44756cefbf2d7235ae4 WatchSource:0}: Error finding container 395f063817b2601473a4c33a6a357311bafefa3a918ec44756cefbf2d7235ae4: Status 404 returned error can't find the container with id 395f063817b2601473a4c33a6a357311bafefa3a918ec44756cefbf2d7235ae4
Sep 12 01:01:45.592712 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:45.592638    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65f9c65e-6611-4ca9-a12a-3814225ed0a8" path="/var/lib/kubelet/pods/65f9c65e-6611-4ca9-a12a-3814225ed0a8/volumes"
Sep 12 01:01:45.593063 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:45.593048    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="992472de-ec76-4916-aaa4-c9935b65d25f" path="/var/lib/kubelet/pods/992472de-ec76-4916-aaa4-c9935b65d25f/volumes"
Sep 12 01:01:45.610122 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:45.609967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq" event={"ID":"dbcdc926-82c2-41d7-a231-81280d5d7864","Type":"ContainerStarted","Data":"4def96c38eca5d3e72bc34c5eafca1df505f85679f7d1ca63c9953b01bdb974d"}
Sep 12 01:01:45.610122 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:45.610024    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq" event={"ID":"dbcdc926-82c2-41d7-a231-81280d5d7864","Type":"ContainerStarted","Data":"00cdc57e51942f415f59ce9ba49149de894d5a8b0455a6e96696664b0c6f4bd7"}
Sep 12 01:01:45.610122 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:45.610060    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"
Sep 12 01:01:45.611759 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:45.611734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879" event={"ID":"e9aeea73-045e-4e21-b0ce-644cf60649ad","Type":"ContainerStarted","Data":"b3b5cbfba5c325c933d0e192fe1bcc37916a01aa487dd62f814b57749df81956"}
Sep 12 01:01:45.611893 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:45.611780    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879" event={"ID":"e9aeea73-045e-4e21-b0ce-644cf60649ad","Type":"ContainerStarted","Data":"395f063817b2601473a4c33a6a357311bafefa3a918ec44756cefbf2d7235ae4"}
Sep 12 01:01:45.611893 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:45.611823    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"
Sep 12 01:01:45.626663 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:45.626620    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq" podStartSLOduration=1.626588161 podStartE2EDuration="1.626588161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:01:45.625362799 +0000 UTC m=+1130.544654588" watchObservedRunningTime="2026-09-12 01:01:45.626588161 +0000 UTC m=+1130.545879972"
Sep 12 01:01:45.641949 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:45.641910    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879" podStartSLOduration=1.6418973860000001 podStartE2EDuration="1.641897386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:01:45.640670425 +0000 UTC m=+1130.559962215" watchObservedRunningTime="2026-09-12 01:01:45.641897386 +0000 UTC m=+1130.561189174"
Sep 12 01:01:49.512239 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:49.512203    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"]
Sep 12 01:01:49.512738 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:49.512415    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj" podUID="a5093418-eae0-4eaa-bb97-767e71de230e" containerName="istio-proxy" containerID="cri-o://b110bd7a6ac48479173a5fb04038e820656fe39853e8170609a87e01acb62062" gracePeriod=30
Sep 12 01:01:54.670067 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.670036    2579 generic.go:356] "Generic (PLEG): container finished" podID="a5093418-eae0-4eaa-bb97-767e71de230e" containerID="b110bd7a6ac48479173a5fb04038e820656fe39853e8170609a87e01acb62062" exitCode=0
Sep 12 01:01:54.670455 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.670105    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj" event={"ID":"a5093418-eae0-4eaa-bb97-767e71de230e","Type":"ContainerDied","Data":"b110bd7a6ac48479173a5fb04038e820656fe39853e8170609a87e01acb62062"}
Sep 12 01:01:54.670455 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.670177    2579 scope.go:117] "RemoveContainer" containerID="092595da3092db66e2e6d72f85bd70aae095f77d25bc04964af334a79e7f0280"
Sep 12 01:01:54.766663 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.766638    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 01:01:54.943700 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.943671    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5093418-eae0-4eaa-bb97-767e71de230e-istio-token\") pod \"a5093418-eae0-4eaa-bb97-767e71de230e\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") "
Sep 12 01:01:54.943877 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.943712    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-workload-certs\") pod \"a5093418-eae0-4eaa-bb97-767e71de230e\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") "
Sep 12 01:01:54.943877 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.943730    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5093418-eae0-4eaa-bb97-767e71de230e-istiod-ca-cert\") pod \"a5093418-eae0-4eaa-bb97-767e71de230e\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") "
Sep 12 01:01:54.943877 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.943760    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-istio-envoy\") pod \"a5093418-eae0-4eaa-bb97-767e71de230e\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") "
Sep 12 01:01:54.943877 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.943787    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5093418-eae0-4eaa-bb97-767e71de230e-istio-podinfo\") pod \"a5093418-eae0-4eaa-bb97-767e71de230e\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") "
Sep 12 01:01:54.943877 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.943815    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwphr\" (UniqueName: \"kubernetes.io/projected/a5093418-eae0-4eaa-bb97-767e71de230e-kube-api-access-vwphr\") pod \"a5093418-eae0-4eaa-bb97-767e71de230e\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") "
Sep 12 01:01:54.943877 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.943840    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-credential-socket\") pod \"a5093418-eae0-4eaa-bb97-767e71de230e\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") "
Sep 12 01:01:54.943877 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.943879    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-istio-data\") pod \"a5093418-eae0-4eaa-bb97-767e71de230e\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") "
Sep 12 01:01:54.944233 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.943953    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-workload-socket\") pod \"a5093418-eae0-4eaa-bb97-767e71de230e\" (UID: \"a5093418-eae0-4eaa-bb97-767e71de230e\") "
Sep 12 01:01:54.944233 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.944029    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a5093418-eae0-4eaa-bb97-767e71de230e" (UID: "a5093418-eae0-4eaa-bb97-767e71de230e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:01:54.944233 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.944078    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5093418-eae0-4eaa-bb97-767e71de230e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a5093418-eae0-4eaa-bb97-767e71de230e" (UID: "a5093418-eae0-4eaa-bb97-767e71de230e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:01:54.944386 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.944278    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a5093418-eae0-4eaa-bb97-767e71de230e" (UID: "a5093418-eae0-4eaa-bb97-767e71de230e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:01:54.944386 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.944288    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:54.944386 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.944293    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a5093418-eae0-4eaa-bb97-767e71de230e" (UID: "a5093418-eae0-4eaa-bb97-767e71de230e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:01:54.944386 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.944309    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5093418-eae0-4eaa-bb97-767e71de230e-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:54.944386 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.944372    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-istio-data" (OuterVolumeSpecName: "istio-data") pod "a5093418-eae0-4eaa-bb97-767e71de230e" (UID: "a5093418-eae0-4eaa-bb97-767e71de230e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:01:54.946364 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.946345    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5093418-eae0-4eaa-bb97-767e71de230e-istio-token" (OuterVolumeSpecName: "istio-token") pod "a5093418-eae0-4eaa-bb97-767e71de230e" (UID: "a5093418-eae0-4eaa-bb97-767e71de230e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:01:54.946438 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.946359    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5093418-eae0-4eaa-bb97-767e71de230e-kube-api-access-vwphr" (OuterVolumeSpecName: "kube-api-access-vwphr") pod "a5093418-eae0-4eaa-bb97-767e71de230e" (UID: "a5093418-eae0-4eaa-bb97-767e71de230e"). InnerVolumeSpecName "kube-api-access-vwphr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:01:54.946438 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.946397    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a5093418-eae0-4eaa-bb97-767e71de230e" (UID: "a5093418-eae0-4eaa-bb97-767e71de230e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:01:54.946438 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:54.946397    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5093418-eae0-4eaa-bb97-767e71de230e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a5093418-eae0-4eaa-bb97-767e71de230e" (UID: "a5093418-eae0-4eaa-bb97-767e71de230e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 01:01:55.045408 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:55.045378    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwphr\" (UniqueName: \"kubernetes.io/projected/a5093418-eae0-4eaa-bb97-767e71de230e-kube-api-access-vwphr\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:55.045408 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:55.045404    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:55.045570 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:55.045415    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:55.045570 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:55.045424    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:55.045570 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:55.045432    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5093418-eae0-4eaa-bb97-767e71de230e-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:55.045570 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:55.045441    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5093418-eae0-4eaa-bb97-767e71de230e-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:55.045570 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:55.045449    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5093418-eae0-4eaa-bb97-767e71de230e-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:01:55.676262 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:55.676240    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"
Sep 12 01:01:55.676558 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:55.676240    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj" event={"ID":"a5093418-eae0-4eaa-bb97-767e71de230e","Type":"ContainerDied","Data":"6acc121580fc3d583dac666bb269e53bd4f92d5f9e004f87a0603db49101fac3"}
Sep 12 01:01:55.676558 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:55.676370    2579 scope.go:117] "RemoveContainer" containerID="b110bd7a6ac48479173a5fb04038e820656fe39853e8170609a87e01acb62062"
Sep 12 01:01:55.695188 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:55.695164    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"]
Sep 12 01:01:55.701242 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:55.701219    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-8a4fbb-openshift-default-7449d76d7b-vw4wj"]
Sep 12 01:01:56.619046 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:56.619015    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"
Sep 12 01:01:56.619202 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:56.619147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"
Sep 12 01:01:57.591761 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:01:57.591729    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5093418-eae0-4eaa-bb97-767e71de230e" path="/var/lib/kubelet/pods/a5093418-eae0-4eaa-bb97-767e71de230e/volumes"
Sep 12 01:02:03.026770 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.026733    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4"]
Sep 12 01:02:03.027182 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.027167    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5093418-eae0-4eaa-bb97-767e71de230e" containerName="istio-proxy"
Sep 12 01:02:03.027182 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.027180    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5093418-eae0-4eaa-bb97-767e71de230e" containerName="istio-proxy"
Sep 12 01:02:03.027270 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.027189    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5093418-eae0-4eaa-bb97-767e71de230e" containerName="istio-proxy"
Sep 12 01:02:03.027270 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.027195    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5093418-eae0-4eaa-bb97-767e71de230e" containerName="istio-proxy"
Sep 12 01:02:03.027270 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.027249    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5093418-eae0-4eaa-bb97-767e71de230e" containerName="istio-proxy"
Sep 12 01:02:03.027270 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.027258    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5093418-eae0-4eaa-bb97-767e71de230e" containerName="istio-proxy"
Sep 12 01:02:03.031377 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.031358    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4"
Sep 12 01:02:03.034393 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.034372    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:02:03.034520 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.034438    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2rn9k\""
Sep 12 01:02:03.041294 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.041211    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4"]
Sep 12 01:02:03.120532 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.120497    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd7023a-5426-4568-a994-8e8b8728c769-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4\" (UID: \"5cd7023a-5426-4568-a994-8e8b8728c769\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4"
Sep 12 01:02:03.221385 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.221357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd7023a-5426-4568-a994-8e8b8728c769-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4\" (UID: \"5cd7023a-5426-4568-a994-8e8b8728c769\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4"
Sep 12 01:02:03.222085 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.222063    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd7023a-5426-4568-a994-8e8b8728c769-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4\" (UID: \"5cd7023a-5426-4568-a994-8e8b8728c769\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4"
Sep 12 01:02:03.341647 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.341546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4"
Sep 12 01:02:03.614661 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.614639    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4"]
Sep 12 01:02:03.616975 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:02:03.616946    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd7023a_5426_4568_a994_8e8b8728c769.slice/crio-f242c26b1ad90b0195fe5c80343108f3ffd0be912c4308d034566fc0cba75a9c WatchSource:0}: Error finding container f242c26b1ad90b0195fe5c80343108f3ffd0be912c4308d034566fc0cba75a9c: Status 404 returned error can't find the container with id f242c26b1ad90b0195fe5c80343108f3ffd0be912c4308d034566fc0cba75a9c
Sep 12 01:02:03.726093 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:03.726063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4" event={"ID":"5cd7023a-5426-4568-a994-8e8b8728c769","Type":"ContainerStarted","Data":"f242c26b1ad90b0195fe5c80343108f3ffd0be912c4308d034566fc0cba75a9c"}
Sep 12 01:02:04.732986 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:04.732952    2579 generic.go:356] "Generic (PLEG): container finished" podID="5cd7023a-5426-4568-a994-8e8b8728c769" containerID="25a38b85026dd71ccef0b0bc4127013f6c21d1f584565ec83a322f760bd77aaa" exitCode=0
Sep 12 01:02:04.733382 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:04.733036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4" event={"ID":"5cd7023a-5426-4568-a994-8e8b8728c769","Type":"ContainerDied","Data":"25a38b85026dd71ccef0b0bc4127013f6c21d1f584565ec83a322f760bd77aaa"}
Sep 12 01:02:05.875060 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:05.875037    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4"
Sep 12 01:02:05.940429 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:05.940394    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd7023a-5426-4568-a994-8e8b8728c769-maas-api-service-ca\") pod \"5cd7023a-5426-4568-a994-8e8b8728c769\" (UID: \"5cd7023a-5426-4568-a994-8e8b8728c769\") "
Sep 12 01:02:05.940801 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:05.940780    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd7023a-5426-4568-a994-8e8b8728c769-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5cd7023a-5426-4568-a994-8e8b8728c769" (UID: "5cd7023a-5426-4568-a994-8e8b8728c769"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:02:06.041677 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:06.041589    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd7023a-5426-4568-a994-8e8b8728c769-maas-api-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:06.747884 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:06.747392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4" event={"ID":"5cd7023a-5426-4568-a994-8e8b8728c769","Type":"ContainerDied","Data":"f242c26b1ad90b0195fe5c80343108f3ffd0be912c4308d034566fc0cba75a9c"}
Sep 12 01:02:06.747884 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:06.747477    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f242c26b1ad90b0195fe5c80343108f3ffd0be912c4308d034566fc0cba75a9c"
Sep 12 01:02:06.747884 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:06.747549    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4"
Sep 12 01:02:10.060689 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.060654    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"]
Sep 12 01:02:10.061140 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.061027    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd7023a-5426-4568-a994-8e8b8728c769" containerName="revoke-keys"
Sep 12 01:02:10.061140 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.061038    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd7023a-5426-4568-a994-8e8b8728c769" containerName="revoke-keys"
Sep 12 01:02:10.061140 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.061113    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd7023a-5426-4568-a994-8e8b8728c769" containerName="revoke-keys"
Sep 12 01:02:10.064274 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.064247    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.067217 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.067199    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-18b9d7-openshift-default-dockercfg-t5lmw\""
Sep 12 01:02:10.068731 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.068711    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-token\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.068823 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.068741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-envoy\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.068823 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.068773    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istiod-ca-cert\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.068928 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.068853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-credential-socket\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.068962 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.068934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rkqq\" (UniqueName: \"kubernetes.io/projected/fa0b453e-3223-4a85-956d-d868ea6e4e9f-kube-api-access-5rkqq\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.068962 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.068953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-workload-socket\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.069044 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.068972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-workload-certs\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.069044 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.068988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-data\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.069044 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.069003    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-podinfo\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.080479 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.080455    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"]
Sep 12 01:02:10.170468 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.170433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rkqq\" (UniqueName: \"kubernetes.io/projected/fa0b453e-3223-4a85-956d-d868ea6e4e9f-kube-api-access-5rkqq\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.170468 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.170479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-workload-socket\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.170759 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.170507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-workload-certs\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.170759 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.170533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-data\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.170759 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.170558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-podinfo\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.170759 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.170680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-token\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.170759 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.170711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-envoy\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.170759 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.170739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istiod-ca-cert\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.171059 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.170768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-credential-socket\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.171059 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.170925    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-workload-socket\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.171384 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.171346    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-credential-socket\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.171817 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.171796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-workload-certs\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.172123 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.172095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istiod-ca-cert\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.172233 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.172211    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-data\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.173766 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.173742    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-podinfo\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.174485 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.174461    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-envoy\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.183933 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.183857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-token\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.184615 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.184555    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rkqq\" (UniqueName: \"kubernetes.io/projected/fa0b453e-3223-4a85-956d-d868ea6e4e9f-kube-api-access-5rkqq\") pod \"e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") " pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.374081 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.373989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:10.651713 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.651629    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"]
Sep 12 01:02:10.654595 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:02:10.654568    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0b453e_3223_4a85_956d_d868ea6e4e9f.slice/crio-ebb3c9b5ae7a9545c2e715bc782dc268e29f7540ebb80e681983c872af025bef WatchSource:0}: Error finding container ebb3c9b5ae7a9545c2e715bc782dc268e29f7540ebb80e681983c872af025bef: Status 404 returned error can't find the container with id ebb3c9b5ae7a9545c2e715bc782dc268e29f7540ebb80e681983c872af025bef
Sep 12 01:02:10.656796 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.656761    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 01:02:10.656910 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.656843    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 01:02:10.656910 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.656871    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 01:02:10.768789 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.768744    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m" event={"ID":"fa0b453e-3223-4a85-956d-d868ea6e4e9f","Type":"ContainerStarted","Data":"faed5a5ac487a74d95c01eea8839017c32964c07c6ca361174bbd6d8961ff788"}
Sep 12 01:02:10.768965 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.768821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m" event={"ID":"fa0b453e-3223-4a85-956d-d868ea6e4e9f","Type":"ContainerStarted","Data":"ebb3c9b5ae7a9545c2e715bc782dc268e29f7540ebb80e681983c872af025bef"}
Sep 12 01:02:10.789036 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:10.788990    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m" podStartSLOduration=0.788976931 podStartE2EDuration="788.976931ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:02:10.787981647 +0000 UTC m=+1155.707273456" watchObservedRunningTime="2026-09-12 01:02:10.788976931 +0000 UTC m=+1155.708268717"
Sep 12 01:02:11.374830 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:11.374797    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:11.380037 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:11.380014    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:11.779537 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:11.779497    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:11.780784 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:11.780757    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:15.084885 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.084851    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"]
Sep 12 01:02:15.085443 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.085200    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp" podUID="908baaef-2748-459c-bd57-1f0d994c29bd" containerName="payload-pre-processing" containerID="cri-o://ac8e26db35c5138f1a3955dc76144a8d4591f78c983e1b902ff2715b77141aae" gracePeriod=30
Sep 12 01:02:15.261275 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.261204    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4"]
Sep 12 01:02:15.264321 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.264290    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5cee89-lz7g4"]
Sep 12 01:02:15.341169 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.341113    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"
Sep 12 01:02:15.418752 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.418704    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/908baaef-2748-459c-bd57-1f0d994c29bd-plugins-config-volume\") pod \"908baaef-2748-459c-bd57-1f0d994c29bd\" (UID: \"908baaef-2748-459c-bd57-1f0d994c29bd\") "
Sep 12 01:02:15.418954 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.418852    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc7kr\" (UniqueName: \"kubernetes.io/projected/908baaef-2748-459c-bd57-1f0d994c29bd-kube-api-access-fc7kr\") pod \"908baaef-2748-459c-bd57-1f0d994c29bd\" (UID: \"908baaef-2748-459c-bd57-1f0d994c29bd\") "
Sep 12 01:02:15.419962 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.419930    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908baaef-2748-459c-bd57-1f0d994c29bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "908baaef-2748-459c-bd57-1f0d994c29bd" (UID: "908baaef-2748-459c-bd57-1f0d994c29bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:02:15.421324 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.421295    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908baaef-2748-459c-bd57-1f0d994c29bd-kube-api-access-fc7kr" (OuterVolumeSpecName: "kube-api-access-fc7kr") pod "908baaef-2748-459c-bd57-1f0d994c29bd" (UID: "908baaef-2748-459c-bd57-1f0d994c29bd"). InnerVolumeSpecName "kube-api-access-fc7kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:02:15.519565 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.519532    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/908baaef-2748-459c-bd57-1f0d994c29bd-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:15.519565 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.519558    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc7kr\" (UniqueName: \"kubernetes.io/projected/908baaef-2748-459c-bd57-1f0d994c29bd-kube-api-access-fc7kr\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:15.590786 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.590751    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cd7023a-5426-4568-a994-8e8b8728c769" path="/var/lib/kubelet/pods/5cd7023a-5426-4568-a994-8e8b8728c769/volumes"
Sep 12 01:02:15.801803 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.801761    2579 generic.go:356] "Generic (PLEG): container finished" podID="908baaef-2748-459c-bd57-1f0d994c29bd" containerID="ac8e26db35c5138f1a3955dc76144a8d4591f78c983e1b902ff2715b77141aae" exitCode=0
Sep 12 01:02:15.801997 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.801847    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"
Sep 12 01:02:15.801997 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.801847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp" event={"ID":"908baaef-2748-459c-bd57-1f0d994c29bd","Type":"ContainerDied","Data":"ac8e26db35c5138f1a3955dc76144a8d4591f78c983e1b902ff2715b77141aae"}
Sep 12 01:02:15.801997 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.801916    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp" event={"ID":"908baaef-2748-459c-bd57-1f0d994c29bd","Type":"ContainerDied","Data":"9113c2a4466b7051d1e8379c0fa9641fe0b7ce21265c275b30a4473574610bf8"}
Sep 12 01:02:15.801997 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.801953    2579 scope.go:117] "RemoveContainer" containerID="ac8e26db35c5138f1a3955dc76144a8d4591f78c983e1b902ff2715b77141aae"
Sep 12 01:02:15.811949 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.811934    2579 scope.go:117] "RemoveContainer" containerID="ac8e26db35c5138f1a3955dc76144a8d4591f78c983e1b902ff2715b77141aae"
Sep 12 01:02:15.812198 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:02:15.812180    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac8e26db35c5138f1a3955dc76144a8d4591f78c983e1b902ff2715b77141aae\": container with ID starting with ac8e26db35c5138f1a3955dc76144a8d4591f78c983e1b902ff2715b77141aae not found: ID does not exist" containerID="ac8e26db35c5138f1a3955dc76144a8d4591f78c983e1b902ff2715b77141aae"
Sep 12 01:02:15.812257 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.812207    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac8e26db35c5138f1a3955dc76144a8d4591f78c983e1b902ff2715b77141aae"} err="failed to get container status \"ac8e26db35c5138f1a3955dc76144a8d4591f78c983e1b902ff2715b77141aae\": rpc error: code = NotFound desc = could not find container \"ac8e26db35c5138f1a3955dc76144a8d4591f78c983e1b902ff2715b77141aae\": container with ID starting with ac8e26db35c5138f1a3955dc76144a8d4591f78c983e1b902ff2715b77141aae not found: ID does not exist"
Sep 12 01:02:15.822222 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.822200    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"]
Sep 12 01:02:15.824317 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:15.824298    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5cee89-57b4795f7d-xkczp"]
Sep 12 01:02:17.526172 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.526141    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"]
Sep 12 01:02:17.526545 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.526532    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908baaef-2748-459c-bd57-1f0d994c29bd" containerName="payload-pre-processing"
Sep 12 01:02:17.526589 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.526548    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="908baaef-2748-459c-bd57-1f0d994c29bd" containerName="payload-pre-processing"
Sep 12 01:02:17.526679 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.526621    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="908baaef-2748-459c-bd57-1f0d994c29bd" containerName="payload-pre-processing"
Sep 12 01:02:17.531147 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.531123    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"
Sep 12 01:02:17.533818 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.533800    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-18b9d7-dockercfg-qt5xp\""
Sep 12 01:02:17.533921 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.533895    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-18b9d7\""
Sep 12 01:02:17.542162 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.542141    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"]
Sep 12 01:02:17.558400 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.558376    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"]
Sep 12 01:02:17.561702 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.561685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"
Sep 12 01:02:17.571744 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.571724    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"]
Sep 12 01:02:17.591845 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.591825    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="908baaef-2748-459c-bd57-1f0d994c29bd" path="/var/lib/kubelet/pods/908baaef-2748-459c-bd57-1f0d994c29bd/volumes"
Sep 12 01:02:17.639527 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.639484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-499br\" (UniqueName: \"kubernetes.io/projected/606bd754-eb44-4cb7-85c5-b68ba4115dd1-kube-api-access-499br\") pod \"payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5\" (UID: \"606bd754-eb44-4cb7-85c5-b68ba4115dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"
Sep 12 01:02:17.639730 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.639660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtq2f\" (UniqueName: \"kubernetes.io/projected/e26c7769-35d9-4d19-9f99-273f4193b16c-kube-api-access-vtq2f\") pod \"payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj\" (UID: \"e26c7769-35d9-4d19-9f99-273f4193b16c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"
Sep 12 01:02:17.639802 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.639747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/606bd754-eb44-4cb7-85c5-b68ba4115dd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5\" (UID: \"606bd754-eb44-4cb7-85c5-b68ba4115dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"
Sep 12 01:02:17.639802 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.639773    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26c7769-35d9-4d19-9f99-273f4193b16c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj\" (UID: \"e26c7769-35d9-4d19-9f99-273f4193b16c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"
Sep 12 01:02:17.741101 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.741067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/606bd754-eb44-4cb7-85c5-b68ba4115dd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5\" (UID: \"606bd754-eb44-4cb7-85c5-b68ba4115dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"
Sep 12 01:02:17.741101 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.741103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26c7769-35d9-4d19-9f99-273f4193b16c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj\" (UID: \"e26c7769-35d9-4d19-9f99-273f4193b16c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"
Sep 12 01:02:17.741375 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.741239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-499br\" (UniqueName: \"kubernetes.io/projected/606bd754-eb44-4cb7-85c5-b68ba4115dd1-kube-api-access-499br\") pod \"payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5\" (UID: \"606bd754-eb44-4cb7-85c5-b68ba4115dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"
Sep 12 01:02:17.741375 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.741328    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtq2f\" (UniqueName: \"kubernetes.io/projected/e26c7769-35d9-4d19-9f99-273f4193b16c-kube-api-access-vtq2f\") pod \"payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj\" (UID: \"e26c7769-35d9-4d19-9f99-273f4193b16c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"
Sep 12 01:02:17.741896 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.741874    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/606bd754-eb44-4cb7-85c5-b68ba4115dd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5\" (UID: \"606bd754-eb44-4cb7-85c5-b68ba4115dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"
Sep 12 01:02:17.741896 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.741895    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26c7769-35d9-4d19-9f99-273f4193b16c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj\" (UID: \"e26c7769-35d9-4d19-9f99-273f4193b16c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"
Sep 12 01:02:17.750969 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.750942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtq2f\" (UniqueName: \"kubernetes.io/projected/e26c7769-35d9-4d19-9f99-273f4193b16c-kube-api-access-vtq2f\") pod \"payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj\" (UID: \"e26c7769-35d9-4d19-9f99-273f4193b16c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"
Sep 12 01:02:17.750969 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.750963    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-499br\" (UniqueName: \"kubernetes.io/projected/606bd754-eb44-4cb7-85c5-b68ba4115dd1-kube-api-access-499br\") pod \"payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5\" (UID: \"606bd754-eb44-4cb7-85c5-b68ba4115dd1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"
Sep 12 01:02:17.842167 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.842081    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"
Sep 12 01:02:17.872412 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:17.872344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"
Sep 12 01:02:18.121991 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:18.121963    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"]
Sep 12 01:02:18.123924 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:02:18.123901    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606bd754_eb44_4cb7_85c5_b68ba4115dd1.slice/crio-136fb6b7d522b2db68e9fa5bbb7b6765626b45e027fdd8b5fb89a5432748d294 WatchSource:0}: Error finding container 136fb6b7d522b2db68e9fa5bbb7b6765626b45e027fdd8b5fb89a5432748d294: Status 404 returned error can't find the container with id 136fb6b7d522b2db68e9fa5bbb7b6765626b45e027fdd8b5fb89a5432748d294
Sep 12 01:02:18.159438 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:18.159396    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"]
Sep 12 01:02:18.160335 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:02:18.160311    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26c7769_35d9_4d19_9f99_273f4193b16c.slice/crio-b09d2be8dde8e8731f83ba3178daf3c6a63741af6e8f5cae7e6fe44739328bd0 WatchSource:0}: Error finding container b09d2be8dde8e8731f83ba3178daf3c6a63741af6e8f5cae7e6fe44739328bd0: Status 404 returned error can't find the container with id b09d2be8dde8e8731f83ba3178daf3c6a63741af6e8f5cae7e6fe44739328bd0
Sep 12 01:02:18.825535 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:18.825502    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj" event={"ID":"e26c7769-35d9-4d19-9f99-273f4193b16c","Type":"ContainerStarted","Data":"23b3caeba649a1cafdd8718ca1f77dc84eb0cab72bc1b20f1b35b6d871382612"}
Sep 12 01:02:18.825996 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:18.825559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj" event={"ID":"e26c7769-35d9-4d19-9f99-273f4193b16c","Type":"ContainerStarted","Data":"b09d2be8dde8e8731f83ba3178daf3c6a63741af6e8f5cae7e6fe44739328bd0"}
Sep 12 01:02:18.825996 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:18.825589    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"
Sep 12 01:02:18.827211 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:18.827183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5" event={"ID":"606bd754-eb44-4cb7-85c5-b68ba4115dd1","Type":"ContainerStarted","Data":"182e590a6f9a6f0ba0814877f17cb7483ed0acaf363591f48d41fd1608184a19"}
Sep 12 01:02:18.827340 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:18.827231    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5" event={"ID":"606bd754-eb44-4cb7-85c5-b68ba4115dd1","Type":"ContainerStarted","Data":"136fb6b7d522b2db68e9fa5bbb7b6765626b45e027fdd8b5fb89a5432748d294"}
Sep 12 01:02:18.827340 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:18.827266    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"
Sep 12 01:02:18.848732 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:18.848687    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj" podStartSLOduration=1.8486751049999999 podStartE2EDuration="1.848675105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:02:18.846172047 +0000 UTC m=+1163.765463836" watchObservedRunningTime="2026-09-12 01:02:18.848675105 +0000 UTC m=+1163.767966888"
Sep 12 01:02:18.870859 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:18.870815    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5" podStartSLOduration=1.870797094 podStartE2EDuration="1.870797094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:02:18.86877612 +0000 UTC m=+1163.788067908" watchObservedRunningTime="2026-09-12 01:02:18.870797094 +0000 UTC m=+1163.790088885"
Sep 12 01:02:22.124454 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:22.124412    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"]
Sep 12 01:02:22.125243 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:22.125209    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7" podUID="7dd61045-fc5a-46f4-adf5-50d41b74a5a6" containerName="istio-proxy" containerID="cri-o://e453c44dd91bec7f49d41dd257c3942d42ed830e722d54668db9168631b07b55" gracePeriod=30
Sep 12 01:02:27.379748 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.379722    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 01:02:27.429429 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.429397    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-data\") pod \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") "
Sep 12 01:02:27.429631 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.429450    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-envoy\") pod \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") "
Sep 12 01:02:27.429631 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.429476    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-workload-socket\") pod \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") "
Sep 12 01:02:27.429631 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.429523    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istiod-ca-cert\") pod \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") "
Sep 12 01:02:27.429631 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.429547    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-workload-certs\") pod \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") "
Sep 12 01:02:27.429631 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.429576    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-credential-socket\") pod \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") "
Sep 12 01:02:27.429908 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.429643    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-token\") pod \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") "
Sep 12 01:02:27.429908 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.429714    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-data" (OuterVolumeSpecName: "istio-data") pod "7dd61045-fc5a-46f4-adf5-50d41b74a5a6" (UID: "7dd61045-fc5a-46f4-adf5-50d41b74a5a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:02:27.429908 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.429745    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh7wk\" (UniqueName: \"kubernetes.io/projected/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-kube-api-access-rh7wk\") pod \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") "
Sep 12 01:02:27.429908 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.429751    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7dd61045-fc5a-46f4-adf5-50d41b74a5a6" (UID: "7dd61045-fc5a-46f4-adf5-50d41b74a5a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:02:27.429908 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.429776    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-podinfo\") pod \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\" (UID: \"7dd61045-fc5a-46f4-adf5-50d41b74a5a6\") "
Sep 12 01:02:27.430152 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.429946    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7dd61045-fc5a-46f4-adf5-50d41b74a5a6" (UID: "7dd61045-fc5a-46f4-adf5-50d41b74a5a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:02:27.430152 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.429960    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7dd61045-fc5a-46f4-adf5-50d41b74a5a6" (UID: "7dd61045-fc5a-46f4-adf5-50d41b74a5a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:02:27.430152 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.430072    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:27.430152 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.430089    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:27.430152 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.430103    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:27.430152 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.430117    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:27.430577 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.430535    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7dd61045-fc5a-46f4-adf5-50d41b74a5a6" (UID: "7dd61045-fc5a-46f4-adf5-50d41b74a5a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:02:27.432449 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.432425    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7dd61045-fc5a-46f4-adf5-50d41b74a5a6" (UID: "7dd61045-fc5a-46f4-adf5-50d41b74a5a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:02:27.432533 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.432456    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-token" (OuterVolumeSpecName: "istio-token") pod "7dd61045-fc5a-46f4-adf5-50d41b74a5a6" (UID: "7dd61045-fc5a-46f4-adf5-50d41b74a5a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:02:27.432533 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.432456    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-kube-api-access-rh7wk" (OuterVolumeSpecName: "kube-api-access-rh7wk") pod "7dd61045-fc5a-46f4-adf5-50d41b74a5a6" (UID: "7dd61045-fc5a-46f4-adf5-50d41b74a5a6"). InnerVolumeSpecName "kube-api-access-rh7wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:02:27.432941 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.432917    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7dd61045-fc5a-46f4-adf5-50d41b74a5a6" (UID: "7dd61045-fc5a-46f4-adf5-50d41b74a5a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 01:02:27.531041 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.531014    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh7wk\" (UniqueName: \"kubernetes.io/projected/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-kube-api-access-rh7wk\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:27.531041 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.531039    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:27.531210 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.531050    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:27.531210 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.531060    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:27.531210 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.531067    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dd61045-fc5a-46f4-adf5-50d41b74a5a6-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:27.875467 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.875376    2579 generic.go:356] "Generic (PLEG): container finished" podID="7dd61045-fc5a-46f4-adf5-50d41b74a5a6" containerID="e453c44dd91bec7f49d41dd257c3942d42ed830e722d54668db9168631b07b55" exitCode=0
Sep 12 01:02:27.875467 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.875449    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"
Sep 12 01:02:27.875689 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.875461    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7" event={"ID":"7dd61045-fc5a-46f4-adf5-50d41b74a5a6","Type":"ContainerDied","Data":"e453c44dd91bec7f49d41dd257c3942d42ed830e722d54668db9168631b07b55"}
Sep 12 01:02:27.875689 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.875524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7" event={"ID":"7dd61045-fc5a-46f4-adf5-50d41b74a5a6","Type":"ContainerDied","Data":"80e35f443ecd978b1bfde7432f24db6b5e695455a92fbd108177eac01f85424d"}
Sep 12 01:02:27.875689 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.875549    2579 scope.go:117] "RemoveContainer" containerID="e453c44dd91bec7f49d41dd257c3942d42ed830e722d54668db9168631b07b55"
Sep 12 01:02:27.884557 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.884528    2579 scope.go:117] "RemoveContainer" containerID="5bfb7cc88ad20236757040a0d179722cf3f6c65d0ef544c635be3dc7b64c8d18"
Sep 12 01:02:27.892700 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.892658    2579 scope.go:117] "RemoveContainer" containerID="e453c44dd91bec7f49d41dd257c3942d42ed830e722d54668db9168631b07b55"
Sep 12 01:02:27.892986 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:02:27.892966    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e453c44dd91bec7f49d41dd257c3942d42ed830e722d54668db9168631b07b55\": container with ID starting with e453c44dd91bec7f49d41dd257c3942d42ed830e722d54668db9168631b07b55 not found: ID does not exist" containerID="e453c44dd91bec7f49d41dd257c3942d42ed830e722d54668db9168631b07b55"
Sep 12 01:02:27.893044 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.892995    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e453c44dd91bec7f49d41dd257c3942d42ed830e722d54668db9168631b07b55"} err="failed to get container status \"e453c44dd91bec7f49d41dd257c3942d42ed830e722d54668db9168631b07b55\": rpc error: code = NotFound desc = could not find container \"e453c44dd91bec7f49d41dd257c3942d42ed830e722d54668db9168631b07b55\": container with ID starting with e453c44dd91bec7f49d41dd257c3942d42ed830e722d54668db9168631b07b55 not found: ID does not exist"
Sep 12 01:02:27.893044 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.893012    2579 scope.go:117] "RemoveContainer" containerID="5bfb7cc88ad20236757040a0d179722cf3f6c65d0ef544c635be3dc7b64c8d18"
Sep 12 01:02:27.893262 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:02:27.893234    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bfb7cc88ad20236757040a0d179722cf3f6c65d0ef544c635be3dc7b64c8d18\": container with ID starting with 5bfb7cc88ad20236757040a0d179722cf3f6c65d0ef544c635be3dc7b64c8d18 not found: ID does not exist" containerID="5bfb7cc88ad20236757040a0d179722cf3f6c65d0ef544c635be3dc7b64c8d18"
Sep 12 01:02:27.893305 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.893263    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bfb7cc88ad20236757040a0d179722cf3f6c65d0ef544c635be3dc7b64c8d18"} err="failed to get container status \"5bfb7cc88ad20236757040a0d179722cf3f6c65d0ef544c635be3dc7b64c8d18\": rpc error: code = NotFound desc = could not find container \"5bfb7cc88ad20236757040a0d179722cf3f6c65d0ef544c635be3dc7b64c8d18\": container with ID starting with 5bfb7cc88ad20236757040a0d179722cf3f6c65d0ef544c635be3dc7b64c8d18 not found: ID does not exist"
Sep 12 01:02:27.893402 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.893384    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"]
Sep 12 01:02:27.899337 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:27.899317    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5cee89-openshift-default-999db547b-mdnc7"]
Sep 12 01:02:29.592128 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:29.592097    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dd61045-fc5a-46f4-adf5-50d41b74a5a6" path="/var/lib/kubelet/pods/7dd61045-fc5a-46f4-adf5-50d41b74a5a6/volumes"
Sep 12 01:02:29.834819 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:29.834789    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"
Sep 12 01:02:29.835019 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:29.835004    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"
Sep 12 01:02:33.030441 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.030368    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bf547a-29819580-zflwx"]
Sep 12 01:02:33.033658 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.033631    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-bf547a-29819580-zflwx"]
Sep 12 01:02:33.060898 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.060867    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"]
Sep 12 01:02:33.061217 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.061158    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889" podUID="b7852365-9fc3-416e-b626-a9f421d56089" containerName="payload-processing" containerID="cri-o://ea0e4916e5ad1a8866adff6cdb8fda16ca826563c58bb91e241f228e0d6a1558" gracePeriod=30
Sep 12 01:02:33.311594 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.311569    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"
Sep 12 01:02:33.382543 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.382504    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7852365-9fc3-416e-b626-a9f421d56089-plugins-config-volume\") pod \"b7852365-9fc3-416e-b626-a9f421d56089\" (UID: \"b7852365-9fc3-416e-b626-a9f421d56089\") "
Sep 12 01:02:33.382735 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.382551    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvzp6\" (UniqueName: \"kubernetes.io/projected/b7852365-9fc3-416e-b626-a9f421d56089-kube-api-access-kvzp6\") pod \"b7852365-9fc3-416e-b626-a9f421d56089\" (UID: \"b7852365-9fc3-416e-b626-a9f421d56089\") "
Sep 12 01:02:33.382971 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.382937    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7852365-9fc3-416e-b626-a9f421d56089-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7852365-9fc3-416e-b626-a9f421d56089" (UID: "b7852365-9fc3-416e-b626-a9f421d56089"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:02:33.389184 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.387581    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7852365-9fc3-416e-b626-a9f421d56089-kube-api-access-kvzp6" (OuterVolumeSpecName: "kube-api-access-kvzp6") pod "b7852365-9fc3-416e-b626-a9f421d56089" (UID: "b7852365-9fc3-416e-b626-a9f421d56089"). InnerVolumeSpecName "kube-api-access-kvzp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:02:33.483792 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.483760    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7852365-9fc3-416e-b626-a9f421d56089-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:33.483792 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.483786    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvzp6\" (UniqueName: \"kubernetes.io/projected/b7852365-9fc3-416e-b626-a9f421d56089-kube-api-access-kvzp6\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:02:33.591243 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.591169    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d4dea0-5119-4846-81e7-7603120d4962" path="/var/lib/kubelet/pods/c7d4dea0-5119-4846-81e7-7603120d4962/volumes"
Sep 12 01:02:33.911526 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.911437    2579 generic.go:356] "Generic (PLEG): container finished" podID="b7852365-9fc3-416e-b626-a9f421d56089" containerID="ea0e4916e5ad1a8866adff6cdb8fda16ca826563c58bb91e241f228e0d6a1558" exitCode=0
Sep 12 01:02:33.911526 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.911482    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"
Sep 12 01:02:33.911765 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.911523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889" event={"ID":"b7852365-9fc3-416e-b626-a9f421d56089","Type":"ContainerDied","Data":"ea0e4916e5ad1a8866adff6cdb8fda16ca826563c58bb91e241f228e0d6a1558"}
Sep 12 01:02:33.911765 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.911579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889" event={"ID":"b7852365-9fc3-416e-b626-a9f421d56089","Type":"ContainerDied","Data":"37fdf94d3234a98c5eb94c6b2e74517ca5a3ca58e22d65c6d133e71454c591b9"}
Sep 12 01:02:33.911765 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.911625    2579 scope.go:117] "RemoveContainer" containerID="ea0e4916e5ad1a8866adff6cdb8fda16ca826563c58bb91e241f228e0d6a1558"
Sep 12 01:02:33.920110 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.920092    2579 scope.go:117] "RemoveContainer" containerID="ea0e4916e5ad1a8866adff6cdb8fda16ca826563c58bb91e241f228e0d6a1558"
Sep 12 01:02:33.920415 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:02:33.920393    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea0e4916e5ad1a8866adff6cdb8fda16ca826563c58bb91e241f228e0d6a1558\": container with ID starting with ea0e4916e5ad1a8866adff6cdb8fda16ca826563c58bb91e241f228e0d6a1558 not found: ID does not exist" containerID="ea0e4916e5ad1a8866adff6cdb8fda16ca826563c58bb91e241f228e0d6a1558"
Sep 12 01:02:33.920508 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.920422    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea0e4916e5ad1a8866adff6cdb8fda16ca826563c58bb91e241f228e0d6a1558"} err="failed to get container status \"ea0e4916e5ad1a8866adff6cdb8fda16ca826563c58bb91e241f228e0d6a1558\": rpc error: code = NotFound desc = could not find container \"ea0e4916e5ad1a8866adff6cdb8fda16ca826563c58bb91e241f228e0d6a1558\": container with ID starting with ea0e4916e5ad1a8866adff6cdb8fda16ca826563c58bb91e241f228e0d6a1558 not found: ID does not exist"
Sep 12 01:02:33.931003 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.930981    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"]
Sep 12 01:02:33.938751 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:33.938729    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bf547a-6b845bc59c-dl889"]
Sep 12 01:02:35.591406 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:35.591376    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7852365-9fc3-416e-b626-a9f421d56089" path="/var/lib/kubelet/pods/b7852365-9fc3-416e-b626-a9f421d56089/volumes"
Sep 12 01:02:45.975928 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:45.975892    2579 generic.go:356] "Generic (PLEG): container finished" podID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" containerID="faed5a5ac487a74d95c01eea8839017c32964c07c6ca361174bbd6d8961ff788" exitCode=0
Sep 12 01:02:45.976431 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:45.975971    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m" event={"ID":"fa0b453e-3223-4a85-956d-d868ea6e4e9f","Type":"ContainerDied","Data":"faed5a5ac487a74d95c01eea8839017c32964c07c6ca361174bbd6d8961ff788"}
Sep 12 01:02:45.976431 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:45.976382    2579 scope.go:117] "RemoveContainer" containerID="faed5a5ac487a74d95c01eea8839017c32964c07c6ca361174bbd6d8961ff788"
Sep 12 01:02:45.977441 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:45.977408    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 01:02:45.977544 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:45.977492    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 01:02:45.977544 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:45.977537    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 01:02:46.983028 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:46.982993    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m" event={"ID":"fa0b453e-3223-4a85-956d-d868ea6e4e9f","Type":"ContainerStarted","Data":"31eaf5a275a3d0a23eae5925231f7aba5461a86331a95458283b761e5e434ed3"}
Sep 12 01:02:47.375017 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:47.374922    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:47.375168 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:47.375139    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 12 01:02:47.375221 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:47.375202    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m" podUID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 12 01:02:48.374385 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:48.374345    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 12 01:02:48.374873 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:48.374415    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m" podUID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 12 01:02:49.374757 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:49.374721    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 12 01:02:49.375153 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:49.374777    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m" podUID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 12 01:02:50.374934 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:50.374896    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 12 01:02:50.375301 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:50.374956    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m" podUID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 12 01:02:51.374409 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.374369    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 12 01:02:51.374592 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.374429    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m" podUID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 12 01:02:51.765192 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.765163    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"]
Sep 12 01:02:51.765548 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.765540    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd61045-fc5a-46f4-adf5-50d41b74a5a6" containerName="istio-proxy"
Sep 12 01:02:51.765613 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.765551    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd61045-fc5a-46f4-adf5-50d41b74a5a6" containerName="istio-proxy"
Sep 12 01:02:51.765613 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.765558    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd61045-fc5a-46f4-adf5-50d41b74a5a6" containerName="istio-proxy"
Sep 12 01:02:51.765613 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.765563    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd61045-fc5a-46f4-adf5-50d41b74a5a6" containerName="istio-proxy"
Sep 12 01:02:51.765613 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.765576    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7852365-9fc3-416e-b626-a9f421d56089" containerName="payload-processing"
Sep 12 01:02:51.765613 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.765583    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7852365-9fc3-416e-b626-a9f421d56089" containerName="payload-processing"
Sep 12 01:02:51.765845 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.765647    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7852365-9fc3-416e-b626-a9f421d56089" containerName="payload-processing"
Sep 12 01:02:51.765845 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.765656    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd61045-fc5a-46f4-adf5-50d41b74a5a6" containerName="istio-proxy"
Sep 12 01:02:51.765845 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.765663    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd61045-fc5a-46f4-adf5-50d41b74a5a6" containerName="istio-proxy"
Sep 12 01:02:51.769234 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.769214    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"
Sep 12 01:02:51.772787 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.772769    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0cfc70f5\""
Sep 12 01:02:51.781090 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.781069    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"]
Sep 12 01:02:51.838460 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.838427    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"]
Sep 12 01:02:51.841880 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.841863    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"
Sep 12 01:02:51.844580 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.844562    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0cfc70f5-dockercfg-krq96\""
Sep 12 01:02:51.853497 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.853477    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"]
Sep 12 01:02:51.960851 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.960823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be61816c-6070-4ee8-9544-ebf7ee6ef5a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d\" (UID: \"be61816c-6070-4ee8-9544-ebf7ee6ef5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"
Sep 12 01:02:51.961023 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.960865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ed8536-b049-4838-a3e4-88a881a28102-plugins-config-volume\") pod \"payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt\" (UID: \"86ed8536-b049-4838-a3e4-88a881a28102\") " pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"
Sep 12 01:02:51.961023 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.960934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh6ns\" (UniqueName: \"kubernetes.io/projected/86ed8536-b049-4838-a3e4-88a881a28102-kube-api-access-wh6ns\") pod \"payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt\" (UID: \"86ed8536-b049-4838-a3e4-88a881a28102\") " pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"
Sep 12 01:02:51.961023 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:51.960960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffk95\" (UniqueName: \"kubernetes.io/projected/be61816c-6070-4ee8-9544-ebf7ee6ef5a8-kube-api-access-ffk95\") pod \"payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d\" (UID: \"be61816c-6070-4ee8-9544-ebf7ee6ef5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"
Sep 12 01:02:52.062420 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:52.062329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh6ns\" (UniqueName: \"kubernetes.io/projected/86ed8536-b049-4838-a3e4-88a881a28102-kube-api-access-wh6ns\") pod \"payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt\" (UID: \"86ed8536-b049-4838-a3e4-88a881a28102\") " pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"
Sep 12 01:02:52.062420 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:52.062370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffk95\" (UniqueName: \"kubernetes.io/projected/be61816c-6070-4ee8-9544-ebf7ee6ef5a8-kube-api-access-ffk95\") pod \"payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d\" (UID: \"be61816c-6070-4ee8-9544-ebf7ee6ef5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"
Sep 12 01:02:52.062420 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:52.062422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be61816c-6070-4ee8-9544-ebf7ee6ef5a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d\" (UID: \"be61816c-6070-4ee8-9544-ebf7ee6ef5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"
Sep 12 01:02:52.062734 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:52.062442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ed8536-b049-4838-a3e4-88a881a28102-plugins-config-volume\") pod \"payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt\" (UID: \"86ed8536-b049-4838-a3e4-88a881a28102\") " pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"
Sep 12 01:02:52.063113 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:52.063094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ed8536-b049-4838-a3e4-88a881a28102-plugins-config-volume\") pod \"payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt\" (UID: \"86ed8536-b049-4838-a3e4-88a881a28102\") " pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"
Sep 12 01:02:52.063157 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:52.063130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be61816c-6070-4ee8-9544-ebf7ee6ef5a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d\" (UID: \"be61816c-6070-4ee8-9544-ebf7ee6ef5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"
Sep 12 01:02:52.070919 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:52.070893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffk95\" (UniqueName: \"kubernetes.io/projected/be61816c-6070-4ee8-9544-ebf7ee6ef5a8-kube-api-access-ffk95\") pod \"payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d\" (UID: \"be61816c-6070-4ee8-9544-ebf7ee6ef5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"
Sep 12 01:02:52.071044 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:52.070969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh6ns\" (UniqueName: \"kubernetes.io/projected/86ed8536-b049-4838-a3e4-88a881a28102-kube-api-access-wh6ns\") pod \"payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt\" (UID: \"86ed8536-b049-4838-a3e4-88a881a28102\") " pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"
Sep 12 01:02:52.079629 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:52.079586    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"
Sep 12 01:02:52.152719 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:52.152668    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"
Sep 12 01:02:52.374467 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:52.374389    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 12 01:02:52.374467 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:52.374443    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m" podUID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 12 01:02:52.382265 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:52.382239    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"]
Sep 12 01:02:52.384080 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:02:52.384055    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ed8536_b049_4838_a3e4_88a881a28102.slice/crio-b970a96ad0d9d617a417d15481b1f2ee50fdc36ba52c1e453b75c37c44475e88 WatchSource:0}: Error finding container b970a96ad0d9d617a417d15481b1f2ee50fdc36ba52c1e453b75c37c44475e88: Status 404 returned error can't find the container with id b970a96ad0d9d617a417d15481b1f2ee50fdc36ba52c1e453b75c37c44475e88
Sep 12 01:02:52.461498 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:52.461474    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"]
Sep 12 01:02:52.463515 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:02:52.463489    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe61816c_6070_4ee8_9544_ebf7ee6ef5a8.slice/crio-fb41a727211e0b5a23991bee8f1e57e469d52f9ca3091f5569939822b35c1ba8 WatchSource:0}: Error finding container fb41a727211e0b5a23991bee8f1e57e469d52f9ca3091f5569939822b35c1ba8: Status 404 returned error can't find the container with id fb41a727211e0b5a23991bee8f1e57e469d52f9ca3091f5569939822b35c1ba8
Sep 12 01:02:53.019348 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:53.019312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d" event={"ID":"be61816c-6070-4ee8-9544-ebf7ee6ef5a8","Type":"ContainerStarted","Data":"12874166cb6121e848113b7802e3a68f88224789e6bf02932940424e8b051a3c"}
Sep 12 01:02:53.019818 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:53.019372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d" event={"ID":"be61816c-6070-4ee8-9544-ebf7ee6ef5a8","Type":"ContainerStarted","Data":"fb41a727211e0b5a23991bee8f1e57e469d52f9ca3091f5569939822b35c1ba8"}
Sep 12 01:02:53.019818 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:53.019490    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"
Sep 12 01:02:53.021165 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:53.021133    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt" event={"ID":"86ed8536-b049-4838-a3e4-88a881a28102","Type":"ContainerStarted","Data":"65fce4723c7d992446098f4e32a232202ebcca94a7cf81b468da0b5dd39ecb48"}
Sep 12 01:02:53.021322 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:53.021210    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"
Sep 12 01:02:53.021322 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:53.021238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt" event={"ID":"86ed8536-b049-4838-a3e4-88a881a28102","Type":"ContainerStarted","Data":"b970a96ad0d9d617a417d15481b1f2ee50fdc36ba52c1e453b75c37c44475e88"}
Sep 12 01:02:53.040661 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:53.040597    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d" podStartSLOduration=2.040577916 podStartE2EDuration="2.040577916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:02:53.03779687 +0000 UTC m=+1197.957088658" watchObservedRunningTime="2026-09-12 01:02:53.040577916 +0000 UTC m=+1197.959869707"
Sep 12 01:02:53.057987 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:53.057944    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt" podStartSLOduration=2.05793237 podStartE2EDuration="2.05793237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:02:53.056179839 +0000 UTC m=+1197.975471626" watchObservedRunningTime="2026-09-12 01:02:53.05793237 +0000 UTC m=+1197.977224173"
Sep 12 01:02:53.379065 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:53.378991    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:53.379268 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:53.379245    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:53.379897 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:53.379880    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:02:55.584399 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:55.584374    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 01:02:55.585110 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:02:55.585085    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 01:03:04.029091 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:04.029059    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"
Sep 12 01:03:04.029469 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:04.029287    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"
Sep 12 01:03:24.593234 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.593198    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"]
Sep 12 01:03:24.596589 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.596573    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"
Sep 12 01:03:24.599216 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.599195    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-61f73f-dockercfg-xwp8x\""
Sep 12 01:03:24.599345 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.599238    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-61f73f\""
Sep 12 01:03:24.607642 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.607591    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"]
Sep 12 01:03:24.609552 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.609531    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"]
Sep 12 01:03:24.613038 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.613017    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"
Sep 12 01:03:24.625042 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.625020    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"]
Sep 12 01:03:24.753382 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.753348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35321de5-f1d0-47f2-b947-577c12191d0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g\" (UID: \"35321de5-f1d0-47f2-b947-577c12191d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"
Sep 12 01:03:24.753539 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.753393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41bc22c2-5fda-4692-aa5b-ddb8ac02be3c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x\" (UID: \"41bc22c2-5fda-4692-aa5b-ddb8ac02be3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"
Sep 12 01:03:24.753539 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.753416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zl6t\" (UniqueName: \"kubernetes.io/projected/35321de5-f1d0-47f2-b947-577c12191d0d-kube-api-access-5zl6t\") pod \"payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g\" (UID: \"35321de5-f1d0-47f2-b947-577c12191d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"
Sep 12 01:03:24.753539 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.753505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl4zf\" (UniqueName: \"kubernetes.io/projected/41bc22c2-5fda-4692-aa5b-ddb8ac02be3c-kube-api-access-bl4zf\") pod \"payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x\" (UID: \"41bc22c2-5fda-4692-aa5b-ddb8ac02be3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"
Sep 12 01:03:24.854550 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.854472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl4zf\" (UniqueName: \"kubernetes.io/projected/41bc22c2-5fda-4692-aa5b-ddb8ac02be3c-kube-api-access-bl4zf\") pod \"payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x\" (UID: \"41bc22c2-5fda-4692-aa5b-ddb8ac02be3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"
Sep 12 01:03:24.854725 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.854555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35321de5-f1d0-47f2-b947-577c12191d0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g\" (UID: \"35321de5-f1d0-47f2-b947-577c12191d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"
Sep 12 01:03:24.854725 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.854583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41bc22c2-5fda-4692-aa5b-ddb8ac02be3c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x\" (UID: \"41bc22c2-5fda-4692-aa5b-ddb8ac02be3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"
Sep 12 01:03:24.854725 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.854630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zl6t\" (UniqueName: \"kubernetes.io/projected/35321de5-f1d0-47f2-b947-577c12191d0d-kube-api-access-5zl6t\") pod \"payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g\" (UID: \"35321de5-f1d0-47f2-b947-577c12191d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"
Sep 12 01:03:24.855277 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.855257    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35321de5-f1d0-47f2-b947-577c12191d0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g\" (UID: \"35321de5-f1d0-47f2-b947-577c12191d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"
Sep 12 01:03:24.855339 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.855286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41bc22c2-5fda-4692-aa5b-ddb8ac02be3c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x\" (UID: \"41bc22c2-5fda-4692-aa5b-ddb8ac02be3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"
Sep 12 01:03:24.863421 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.863389    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl4zf\" (UniqueName: \"kubernetes.io/projected/41bc22c2-5fda-4692-aa5b-ddb8ac02be3c-kube-api-access-bl4zf\") pod \"payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x\" (UID: \"41bc22c2-5fda-4692-aa5b-ddb8ac02be3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"
Sep 12 01:03:24.863644 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.863622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zl6t\" (UniqueName: \"kubernetes.io/projected/35321de5-f1d0-47f2-b947-577c12191d0d-kube-api-access-5zl6t\") pod \"payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g\" (UID: \"35321de5-f1d0-47f2-b947-577c12191d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"
Sep 12 01:03:24.908246 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.908218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"
Sep 12 01:03:24.924363 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:24.924318    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"
Sep 12 01:03:25.185790 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:25.185761    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"]
Sep 12 01:03:25.188400 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:03:25.188367    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35321de5_f1d0_47f2_b947_577c12191d0d.slice/crio-ab0f87bebcf5437e6d44ee203da50d5927e558956dd0bae7bcab1ced60d20cbd WatchSource:0}: Error finding container ab0f87bebcf5437e6d44ee203da50d5927e558956dd0bae7bcab1ced60d20cbd: Status 404 returned error can't find the container with id ab0f87bebcf5437e6d44ee203da50d5927e558956dd0bae7bcab1ced60d20cbd
Sep 12 01:03:25.201676 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:25.201643    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g" event={"ID":"35321de5-f1d0-47f2-b947-577c12191d0d","Type":"ContainerStarted","Data":"ab0f87bebcf5437e6d44ee203da50d5927e558956dd0bae7bcab1ced60d20cbd"}
Sep 12 01:03:25.208211 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:25.208188    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"]
Sep 12 01:03:25.209881 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:03:25.209859    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41bc22c2_5fda_4692_aa5b_ddb8ac02be3c.slice/crio-746309dfac627feeedc4437d7795d589638f05e88c7dd542dc2d5d9e59a39fca WatchSource:0}: Error finding container 746309dfac627feeedc4437d7795d589638f05e88c7dd542dc2d5d9e59a39fca: Status 404 returned error can't find the container with id 746309dfac627feeedc4437d7795d589638f05e88c7dd542dc2d5d9e59a39fca
Sep 12 01:03:26.209198 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:26.209144    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x" event={"ID":"41bc22c2-5fda-4692-aa5b-ddb8ac02be3c","Type":"ContainerStarted","Data":"2c52604f2c372646b064ae5d700e478756bdf6209f2d115676cac6bc484b3649"}
Sep 12 01:03:26.209656 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:26.209212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x" event={"ID":"41bc22c2-5fda-4692-aa5b-ddb8ac02be3c","Type":"ContainerStarted","Data":"746309dfac627feeedc4437d7795d589638f05e88c7dd542dc2d5d9e59a39fca"}
Sep 12 01:03:26.209656 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:26.209372    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"
Sep 12 01:03:26.211401 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:26.211372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g" event={"ID":"35321de5-f1d0-47f2-b947-577c12191d0d","Type":"ContainerStarted","Data":"ec395f18be853fb063708e079be83801b259ec38c17343799ec35caa6ee948c4"}
Sep 12 01:03:26.211526 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:26.211431    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"
Sep 12 01:03:26.238656 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:26.238593    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x" podStartSLOduration=2.23857826 podStartE2EDuration="2.23857826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:03:26.236731272 +0000 UTC m=+1231.156023060" watchObservedRunningTime="2026-09-12 01:03:26.23857826 +0000 UTC m=+1231.157870045"
Sep 12 01:03:26.270429 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:26.270386    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g" podStartSLOduration=2.270374601 podStartE2EDuration="2.270374601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:03:26.268345328 +0000 UTC m=+1231.187637115" watchObservedRunningTime="2026-09-12 01:03:26.270374601 +0000 UTC m=+1231.189666385"
Sep 12 01:03:37.218734 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:37.218702    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"
Sep 12 01:03:37.219192 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:03:37.218792    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"
Sep 12 01:04:00.007878 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:00.007793    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"]
Sep 12 01:04:00.011362 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:00.011340    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"
Sep 12 01:04:00.014259 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:00.014239    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-ef5997-dockercfg-kctmv\""
Sep 12 01:04:00.014394 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:00.014248    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-ef5997\""
Sep 12 01:04:00.026406 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:00.026383    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"]
Sep 12 01:04:00.061854 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:00.061825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/286ac2d9-4dff-4b4c-8db2-78e0a944c990-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc\" (UID: \"286ac2d9-4dff-4b4c-8db2-78e0a944c990\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"
Sep 12 01:04:00.061980 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:00.061860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j98hn\" (UniqueName: \"kubernetes.io/projected/286ac2d9-4dff-4b4c-8db2-78e0a944c990-kube-api-access-j98hn\") pod \"payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc\" (UID: \"286ac2d9-4dff-4b4c-8db2-78e0a944c990\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"
Sep 12 01:04:00.163282 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:00.163234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/286ac2d9-4dff-4b4c-8db2-78e0a944c990-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc\" (UID: \"286ac2d9-4dff-4b4c-8db2-78e0a944c990\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"
Sep 12 01:04:00.163282 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:00.163279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j98hn\" (UniqueName: \"kubernetes.io/projected/286ac2d9-4dff-4b4c-8db2-78e0a944c990-kube-api-access-j98hn\") pod \"payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc\" (UID: \"286ac2d9-4dff-4b4c-8db2-78e0a944c990\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"
Sep 12 01:04:00.164046 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:00.164027    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/286ac2d9-4dff-4b4c-8db2-78e0a944c990-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc\" (UID: \"286ac2d9-4dff-4b4c-8db2-78e0a944c990\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"
Sep 12 01:04:00.172428 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:00.172402    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j98hn\" (UniqueName: \"kubernetes.io/projected/286ac2d9-4dff-4b4c-8db2-78e0a944c990-kube-api-access-j98hn\") pod \"payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc\" (UID: \"286ac2d9-4dff-4b4c-8db2-78e0a944c990\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"
Sep 12 01:04:00.321361 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:00.321275    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"
Sep 12 01:04:00.597347 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:00.597312    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"]
Sep 12 01:04:00.599375 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:04:00.599344    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286ac2d9_4dff_4b4c_8db2_78e0a944c990.slice/crio-1534a9b239b45355048d2287c2b1db3cf3ffbf6c7898041edde9a7595f49d6f2 WatchSource:0}: Error finding container 1534a9b239b45355048d2287c2b1db3cf3ffbf6c7898041edde9a7595f49d6f2: Status 404 returned error can't find the container with id 1534a9b239b45355048d2287c2b1db3cf3ffbf6c7898041edde9a7595f49d6f2
Sep 12 01:04:01.402764 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:01.402726    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc" event={"ID":"286ac2d9-4dff-4b4c-8db2-78e0a944c990","Type":"ContainerStarted","Data":"2b39f99367d1254bdbb3ab8a64a2af008ab298990137549380ea5b60e9df5589"}
Sep 12 01:04:01.403245 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:01.402790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc" event={"ID":"286ac2d9-4dff-4b4c-8db2-78e0a944c990","Type":"ContainerStarted","Data":"1534a9b239b45355048d2287c2b1db3cf3ffbf6c7898041edde9a7595f49d6f2"}
Sep 12 01:04:01.403245 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:01.402846    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"
Sep 12 01:04:01.419735 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:01.419689    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc" podStartSLOduration=2.419675191 podStartE2EDuration="2.419675191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:04:01.418904918 +0000 UTC m=+1266.338196716" watchObservedRunningTime="2026-09-12 01:04:01.419675191 +0000 UTC m=+1266.338966978"
Sep 12 01:04:12.412067 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:12.412035    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"
Sep 12 01:04:55.686129 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:04:55.686091    2579 scope.go:117] "RemoveContainer" containerID="0583e10bb6f7eabc7ca4ed6d9fe344cd0571cc06d7b79022584541a8bf4b2143"
Sep 12 01:05:11.550063 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.550019    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"]
Sep 12 01:05:11.554847 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.554827    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.558267 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.558244    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-61f73f\"/\"openshift-service-ca.crt\""
Sep 12 01:05:11.558267 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.558255    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-61f73f\"/\"default-dockercfg-c4brc\""
Sep 12 01:05:11.559390 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.559372    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-61f73f\"/\"auth-test-model-61f73f-kserve-self-signed-certs\""
Sep 12 01:05:11.559458 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.559399    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-61f73f\"/\"kube-root-ca.crt\""
Sep 12 01:05:11.574231 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.574200    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"]
Sep 12 01:05:11.588582 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.588563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-dshm\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.588715 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.588590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-tmp-dir\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.588715 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.588627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnfjm\" (UniqueName: \"kubernetes.io/projected/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-kube-api-access-dnfjm\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.588840 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.588731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-home\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.588840 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.588767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-tls-certs\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.588944 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.588862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-model-cache\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.689625 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.689567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-dshm\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.689792 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.689633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-tmp-dir\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.689792 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.689670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnfjm\" (UniqueName: \"kubernetes.io/projected/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-kube-api-access-dnfjm\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.689792 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.689719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-home\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.689792 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.689748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-tls-certs\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.689996 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.689837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-model-cache\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.690198 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.690173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-home\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.690282 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.690205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-model-cache\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.690282 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.690266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-tmp-dir\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.696186 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.692195    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-dshm\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.696186 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.692748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-tls-certs\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.704952 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.704924    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnfjm\" (UniqueName: \"kubernetes.io/projected/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-kube-api-access-dnfjm\") pod \"auth-test-model-61f73f-kserve-684cd59855-dwf8l\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") " pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:11.864254 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:11.864170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:12.136916 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:12.136892    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"]
Sep 12 01:05:12.138996 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:05:12.138958    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf9f0c29_9c94_47c2_bc4e_09ad14d3041e.slice/crio-ef191c62b61ea710b7d63097773374e9a48576447cf4e87b3b982f4accb6dd6b WatchSource:0}: Error finding container ef191c62b61ea710b7d63097773374e9a48576447cf4e87b3b982f4accb6dd6b: Status 404 returned error can't find the container with id ef191c62b61ea710b7d63097773374e9a48576447cf4e87b3b982f4accb6dd6b
Sep 12 01:05:12.779148 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:12.779118    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l" event={"ID":"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e","Type":"ContainerStarted","Data":"6b05cb463042a2d756c62db149db9a1150af1e92d97a5e9fb076f83c4c8c2692"}
Sep 12 01:05:12.779148 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:12.779157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l" event={"ID":"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e","Type":"ContainerStarted","Data":"ef191c62b61ea710b7d63097773374e9a48576447cf4e87b3b982f4accb6dd6b"}
Sep 12 01:05:12.796922 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:12.796870    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l" podStartSLOduration=1.796851453 podStartE2EDuration="1.796851453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:05:12.796298694 +0000 UTC m=+1337.715590480" watchObservedRunningTime="2026-09-12 01:05:12.796851453 +0000 UTC m=+1337.716143241"
Sep 12 01:05:21.864748 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:21.864668    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:21.864748 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:21.864717    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:21.877241 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:21.877221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:22.844821 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:22.844794    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:05:44.908870 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:44.908830    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"]
Sep 12 01:05:44.909475 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:44.909080    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt" podUID="86ed8536-b049-4838-a3e4-88a881a28102" containerName="payload-processing" containerID="cri-o://65fce4723c7d992446098f4e32a232202ebcca94a7cf81b468da0b5dd39ecb48" gracePeriod=30
Sep 12 01:05:44.915952 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:44.915922    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"]
Sep 12 01:05:44.916380 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:44.916147    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d" podUID="be61816c-6070-4ee8-9544-ebf7ee6ef5a8" containerName="payload-pre-processing" containerID="cri-o://12874166cb6121e848113b7802e3a68f88224789e6bf02932940424e8b051a3c" gracePeriod=30
Sep 12 01:05:45.222168 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.222136    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"
Sep 12 01:05:45.225385 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.225360    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"
Sep 12 01:05:45.285509 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.285482    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh6ns\" (UniqueName: \"kubernetes.io/projected/86ed8536-b049-4838-a3e4-88a881a28102-kube-api-access-wh6ns\") pod \"86ed8536-b049-4838-a3e4-88a881a28102\" (UID: \"86ed8536-b049-4838-a3e4-88a881a28102\") "
Sep 12 01:05:45.285684 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.285551    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ed8536-b049-4838-a3e4-88a881a28102-plugins-config-volume\") pod \"86ed8536-b049-4838-a3e4-88a881a28102\" (UID: \"86ed8536-b049-4838-a3e4-88a881a28102\") "
Sep 12 01:05:45.285974 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.285933    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ed8536-b049-4838-a3e4-88a881a28102-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86ed8536-b049-4838-a3e4-88a881a28102" (UID: "86ed8536-b049-4838-a3e4-88a881a28102"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:05:45.287706 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.287681    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ed8536-b049-4838-a3e4-88a881a28102-kube-api-access-wh6ns" (OuterVolumeSpecName: "kube-api-access-wh6ns") pod "86ed8536-b049-4838-a3e4-88a881a28102" (UID: "86ed8536-b049-4838-a3e4-88a881a28102"). InnerVolumeSpecName "kube-api-access-wh6ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:05:45.386209 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.386179    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be61816c-6070-4ee8-9544-ebf7ee6ef5a8-plugins-config-volume\") pod \"be61816c-6070-4ee8-9544-ebf7ee6ef5a8\" (UID: \"be61816c-6070-4ee8-9544-ebf7ee6ef5a8\") "
Sep 12 01:05:45.386365 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.386249    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffk95\" (UniqueName: \"kubernetes.io/projected/be61816c-6070-4ee8-9544-ebf7ee6ef5a8-kube-api-access-ffk95\") pod \"be61816c-6070-4ee8-9544-ebf7ee6ef5a8\" (UID: \"be61816c-6070-4ee8-9544-ebf7ee6ef5a8\") "
Sep 12 01:05:45.386460 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.386450    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh6ns\" (UniqueName: \"kubernetes.io/projected/86ed8536-b049-4838-a3e4-88a881a28102-kube-api-access-wh6ns\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:05:45.386499 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.386462    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86ed8536-b049-4838-a3e4-88a881a28102-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:05:45.386535 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.386503    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be61816c-6070-4ee8-9544-ebf7ee6ef5a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be61816c-6070-4ee8-9544-ebf7ee6ef5a8" (UID: "be61816c-6070-4ee8-9544-ebf7ee6ef5a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:05:45.388233 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.388212    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be61816c-6070-4ee8-9544-ebf7ee6ef5a8-kube-api-access-ffk95" (OuterVolumeSpecName: "kube-api-access-ffk95") pod "be61816c-6070-4ee8-9544-ebf7ee6ef5a8" (UID: "be61816c-6070-4ee8-9544-ebf7ee6ef5a8"). InnerVolumeSpecName "kube-api-access-ffk95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:05:45.487111 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.487041    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffk95\" (UniqueName: \"kubernetes.io/projected/be61816c-6070-4ee8-9544-ebf7ee6ef5a8-kube-api-access-ffk95\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:05:45.487111 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.487065    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be61816c-6070-4ee8-9544-ebf7ee6ef5a8-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:05:45.960929 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.960896    2579 generic.go:356] "Generic (PLEG): container finished" podID="be61816c-6070-4ee8-9544-ebf7ee6ef5a8" containerID="12874166cb6121e848113b7802e3a68f88224789e6bf02932940424e8b051a3c" exitCode=0
Sep 12 01:05:45.961394 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.960938    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"
Sep 12 01:05:45.961394 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.960934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d" event={"ID":"be61816c-6070-4ee8-9544-ebf7ee6ef5a8","Type":"ContainerDied","Data":"12874166cb6121e848113b7802e3a68f88224789e6bf02932940424e8b051a3c"}
Sep 12 01:05:45.961394 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.961066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d" event={"ID":"be61816c-6070-4ee8-9544-ebf7ee6ef5a8","Type":"ContainerDied","Data":"fb41a727211e0b5a23991bee8f1e57e469d52f9ca3091f5569939822b35c1ba8"}
Sep 12 01:05:45.961394 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.961096    2579 scope.go:117] "RemoveContainer" containerID="12874166cb6121e848113b7802e3a68f88224789e6bf02932940424e8b051a3c"
Sep 12 01:05:45.962541 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.962521    2579 generic.go:356] "Generic (PLEG): container finished" podID="86ed8536-b049-4838-a3e4-88a881a28102" containerID="65fce4723c7d992446098f4e32a232202ebcca94a7cf81b468da0b5dd39ecb48" exitCode=0
Sep 12 01:05:45.962709 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.962564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt" event={"ID":"86ed8536-b049-4838-a3e4-88a881a28102","Type":"ContainerDied","Data":"65fce4723c7d992446098f4e32a232202ebcca94a7cf81b468da0b5dd39ecb48"}
Sep 12 01:05:45.962709 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.962570    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"
Sep 12 01:05:45.962709 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.962656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt" event={"ID":"86ed8536-b049-4838-a3e4-88a881a28102","Type":"ContainerDied","Data":"b970a96ad0d9d617a417d15481b1f2ee50fdc36ba52c1e453b75c37c44475e88"}
Sep 12 01:05:45.970397 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.970381    2579 scope.go:117] "RemoveContainer" containerID="12874166cb6121e848113b7802e3a68f88224789e6bf02932940424e8b051a3c"
Sep 12 01:05:45.970696 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:05:45.970674    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12874166cb6121e848113b7802e3a68f88224789e6bf02932940424e8b051a3c\": container with ID starting with 12874166cb6121e848113b7802e3a68f88224789e6bf02932940424e8b051a3c not found: ID does not exist" containerID="12874166cb6121e848113b7802e3a68f88224789e6bf02932940424e8b051a3c"
Sep 12 01:05:45.970759 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.970702    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12874166cb6121e848113b7802e3a68f88224789e6bf02932940424e8b051a3c"} err="failed to get container status \"12874166cb6121e848113b7802e3a68f88224789e6bf02932940424e8b051a3c\": rpc error: code = NotFound desc = could not find container \"12874166cb6121e848113b7802e3a68f88224789e6bf02932940424e8b051a3c\": container with ID starting with 12874166cb6121e848113b7802e3a68f88224789e6bf02932940424e8b051a3c not found: ID does not exist"
Sep 12 01:05:45.970759 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.970719    2579 scope.go:117] "RemoveContainer" containerID="65fce4723c7d992446098f4e32a232202ebcca94a7cf81b468da0b5dd39ecb48"
Sep 12 01:05:45.978589 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.978571    2579 scope.go:117] "RemoveContainer" containerID="65fce4723c7d992446098f4e32a232202ebcca94a7cf81b468da0b5dd39ecb48"
Sep 12 01:05:45.978892 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:05:45.978867    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fce4723c7d992446098f4e32a232202ebcca94a7cf81b468da0b5dd39ecb48\": container with ID starting with 65fce4723c7d992446098f4e32a232202ebcca94a7cf81b468da0b5dd39ecb48 not found: ID does not exist" containerID="65fce4723c7d992446098f4e32a232202ebcca94a7cf81b468da0b5dd39ecb48"
Sep 12 01:05:45.978949 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.978902    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fce4723c7d992446098f4e32a232202ebcca94a7cf81b468da0b5dd39ecb48"} err="failed to get container status \"65fce4723c7d992446098f4e32a232202ebcca94a7cf81b468da0b5dd39ecb48\": rpc error: code = NotFound desc = could not find container \"65fce4723c7d992446098f4e32a232202ebcca94a7cf81b468da0b5dd39ecb48\": container with ID starting with 65fce4723c7d992446098f4e32a232202ebcca94a7cf81b468da0b5dd39ecb48 not found: ID does not exist"
Sep 12 01:05:45.984635 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.984596    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"]
Sep 12 01:05:45.986754 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.986732    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0cfc70f5-5469dd7dc7-4v7vt"]
Sep 12 01:05:45.999035 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:45.999013    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"]
Sep 12 01:05:46.006205 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:46.006182    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0cfc70f5-7cb7f8f598-zjm2d"]
Sep 12 01:05:47.591457 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:47.591429    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86ed8536-b049-4838-a3e4-88a881a28102" path="/var/lib/kubelet/pods/86ed8536-b049-4838-a3e4-88a881a28102/volumes"
Sep 12 01:05:47.591822 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:05:47.591803    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be61816c-6070-4ee8-9544-ebf7ee6ef5a8" path="/var/lib/kubelet/pods/be61816c-6070-4ee8-9544-ebf7ee6ef5a8/volumes"
Sep 12 01:06:55.723860 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:06:55.723789    2579 scope.go:117] "RemoveContainer" containerID="eacc3e264164b5dfb54e6dbfb7f43eb53886c07d3220a5f85ac4042510f74612"
Sep 12 01:06:55.733870 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:06:55.733847    2579 scope.go:117] "RemoveContainer" containerID="4ef57962186e503801a651e5c1a338d53d12a2acb65cb9fb8ec01317cf2dc0bb"
Sep 12 01:07:47.592432 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.592401    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"]
Sep 12 01:07:47.594797 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.592771    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be61816c-6070-4ee8-9544-ebf7ee6ef5a8" containerName="payload-pre-processing"
Sep 12 01:07:47.594797 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.592783    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="be61816c-6070-4ee8-9544-ebf7ee6ef5a8" containerName="payload-pre-processing"
Sep 12 01:07:47.594797 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.592796    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ed8536-b049-4838-a3e4-88a881a28102" containerName="payload-processing"
Sep 12 01:07:47.594797 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.592801    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ed8536-b049-4838-a3e4-88a881a28102" containerName="payload-processing"
Sep 12 01:07:47.594797 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.592858    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="86ed8536-b049-4838-a3e4-88a881a28102" containerName="payload-processing"
Sep 12 01:07:47.594797 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.592870    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="be61816c-6070-4ee8-9544-ebf7ee6ef5a8" containerName="payload-pre-processing"
Sep 12 01:07:47.595673 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.595656    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"
Sep 12 01:07:47.598827 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.598802    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-033ca6\""
Sep 12 01:07:47.601777 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.601754    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"]
Sep 12 01:07:47.624676 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.624651    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"]
Sep 12 01:07:47.630027 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.630002    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"
Sep 12 01:07:47.633860 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.633842    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-033ca6-dockercfg-b9jr7\""
Sep 12 01:07:47.636764 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.636745    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"]
Sep 12 01:07:47.672042 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.672016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szkrx\" (UniqueName: \"kubernetes.io/projected/a90ffb57-a008-4267-b9c9-3609b0b8c8ec-kube-api-access-szkrx\") pod \"payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j\" (UID: \"a90ffb57-a008-4267-b9c9-3609b0b8c8ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"
Sep 12 01:07:47.672170 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.672140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90ffb57-a008-4267-b9c9-3609b0b8c8ec-plugins-config-volume\") pod \"payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j\" (UID: \"a90ffb57-a008-4267-b9c9-3609b0b8c8ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"
Sep 12 01:07:47.672170 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.672159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mffb\" (UniqueName: \"kubernetes.io/projected/a7ed3812-2d25-45a8-b749-9610e5ada035-kube-api-access-8mffb\") pod \"payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm\" (UID: \"a7ed3812-2d25-45a8-b749-9610e5ada035\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"
Sep 12 01:07:47.672302 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.672184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ed3812-2d25-45a8-b749-9610e5ada035-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm\" (UID: \"a7ed3812-2d25-45a8-b749-9610e5ada035\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"
Sep 12 01:07:47.773430 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.773389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90ffb57-a008-4267-b9c9-3609b0b8c8ec-plugins-config-volume\") pod \"payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j\" (UID: \"a90ffb57-a008-4267-b9c9-3609b0b8c8ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"
Sep 12 01:07:47.773430 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.773429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mffb\" (UniqueName: \"kubernetes.io/projected/a7ed3812-2d25-45a8-b749-9610e5ada035-kube-api-access-8mffb\") pod \"payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm\" (UID: \"a7ed3812-2d25-45a8-b749-9610e5ada035\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"
Sep 12 01:07:47.773707 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.773457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ed3812-2d25-45a8-b749-9610e5ada035-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm\" (UID: \"a7ed3812-2d25-45a8-b749-9610e5ada035\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"
Sep 12 01:07:47.773707 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.773492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szkrx\" (UniqueName: \"kubernetes.io/projected/a90ffb57-a008-4267-b9c9-3609b0b8c8ec-kube-api-access-szkrx\") pod \"payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j\" (UID: \"a90ffb57-a008-4267-b9c9-3609b0b8c8ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"
Sep 12 01:07:47.774251 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.774227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ed3812-2d25-45a8-b749-9610e5ada035-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm\" (UID: \"a7ed3812-2d25-45a8-b749-9610e5ada035\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"
Sep 12 01:07:47.774381 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.774358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90ffb57-a008-4267-b9c9-3609b0b8c8ec-plugins-config-volume\") pod \"payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j\" (UID: \"a90ffb57-a008-4267-b9c9-3609b0b8c8ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"
Sep 12 01:07:47.782015 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.781994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szkrx\" (UniqueName: \"kubernetes.io/projected/a90ffb57-a008-4267-b9c9-3609b0b8c8ec-kube-api-access-szkrx\") pod \"payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j\" (UID: \"a90ffb57-a008-4267-b9c9-3609b0b8c8ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"
Sep 12 01:07:47.782113 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.782044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mffb\" (UniqueName: \"kubernetes.io/projected/a7ed3812-2d25-45a8-b749-9610e5ada035-kube-api-access-8mffb\") pod \"payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm\" (UID: \"a7ed3812-2d25-45a8-b749-9610e5ada035\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"
Sep 12 01:07:47.910515 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.910438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"
Sep 12 01:07:47.944203 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:47.944177    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"
Sep 12 01:07:48.210769 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:07:48.210741    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ed3812_2d25_45a8_b749_9610e5ada035.slice/crio-c05a414f569ff596672667a7dd7511a51e8c84aefe5efb62bb99965ccdf277f1 WatchSource:0}: Error finding container c05a414f569ff596672667a7dd7511a51e8c84aefe5efb62bb99965ccdf277f1: Status 404 returned error can't find the container with id c05a414f569ff596672667a7dd7511a51e8c84aefe5efb62bb99965ccdf277f1
Sep 12 01:07:48.210891 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:48.210871    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"]
Sep 12 01:07:48.238755 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:48.238708    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"]
Sep 12 01:07:48.243056 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:07:48.243032    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90ffb57_a008_4267_b9c9_3609b0b8c8ec.slice/crio-4549d9a02f5e2cb19c1164286f6ed05950d591924811515c325f5c7f71a1731b WatchSource:0}: Error finding container 4549d9a02f5e2cb19c1164286f6ed05950d591924811515c325f5c7f71a1731b: Status 404 returned error can't find the container with id 4549d9a02f5e2cb19c1164286f6ed05950d591924811515c325f5c7f71a1731b
Sep 12 01:07:48.598686 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:48.598562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j" event={"ID":"a90ffb57-a008-4267-b9c9-3609b0b8c8ec","Type":"ContainerStarted","Data":"ae7fd6e1876c223a4baad22535fcf8481e4034fa3f2080ac7c22407234c7574d"}
Sep 12 01:07:48.598686 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:48.598642    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j" event={"ID":"a90ffb57-a008-4267-b9c9-3609b0b8c8ec","Type":"ContainerStarted","Data":"4549d9a02f5e2cb19c1164286f6ed05950d591924811515c325f5c7f71a1731b"}
Sep 12 01:07:48.599189 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:48.598840    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"
Sep 12 01:07:48.601067 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:48.601038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm" event={"ID":"a7ed3812-2d25-45a8-b749-9610e5ada035","Type":"ContainerStarted","Data":"7776c433505b7cbacb3ddeb6dd27271149b2b55c0141fb9faea0739d3bc1f7f8"}
Sep 12 01:07:48.601211 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:48.601088    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm" event={"ID":"a7ed3812-2d25-45a8-b749-9610e5ada035","Type":"ContainerStarted","Data":"c05a414f569ff596672667a7dd7511a51e8c84aefe5efb62bb99965ccdf277f1"}
Sep 12 01:07:48.601211 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:48.601117    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"
Sep 12 01:07:48.616509 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:48.616451    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j" podStartSLOduration=1.61643348 podStartE2EDuration="1.61643348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:07:48.615998246 +0000 UTC m=+1493.535290034" watchObservedRunningTime="2026-09-12 01:07:48.61643348 +0000 UTC m=+1493.535725269"
Sep 12 01:07:48.634153 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:48.634098    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm" podStartSLOduration=1.634085786 podStartE2EDuration="1.634085786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:07:48.632212885 +0000 UTC m=+1493.551504697" watchObservedRunningTime="2026-09-12 01:07:48.634085786 +0000 UTC m=+1493.553377602"
Sep 12 01:07:55.617290 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:55.617261    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 01:07:55.617714 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:55.617436    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 01:07:59.610756 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:59.610726    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"
Sep 12 01:07:59.611136 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:07:59.610787    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"
Sep 12 01:08:23.221596 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.221521    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"]
Sep 12 01:08:23.225207 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.225189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"
Sep 12 01:08:23.227994 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.227975    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-49980f\""
Sep 12 01:08:23.228159 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.228140    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-49980f-dockercfg-j5b9d\""
Sep 12 01:08:23.236387 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.236367    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"]
Sep 12 01:08:23.261839 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.261818    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"]
Sep 12 01:08:23.265304 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.265286    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"
Sep 12 01:08:23.275472 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.275453    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"]
Sep 12 01:08:23.395193 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.395161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9377d6cb-e950-40bf-a232-f9646513fdc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22\" (UID: \"9377d6cb-e950-40bf-a232-f9646513fdc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"
Sep 12 01:08:23.395378 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.395216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms4pd\" (UniqueName: \"kubernetes.io/projected/10bd6015-9b3b-45ca-9731-aa5fce0b5f9a-kube-api-access-ms4pd\") pod \"payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m\" (UID: \"10bd6015-9b3b-45ca-9731-aa5fce0b5f9a\") " pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"
Sep 12 01:08:23.395378 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.395245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10bd6015-9b3b-45ca-9731-aa5fce0b5f9a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m\" (UID: \"10bd6015-9b3b-45ca-9731-aa5fce0b5f9a\") " pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"
Sep 12 01:08:23.395471 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.395367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77wmr\" (UniqueName: \"kubernetes.io/projected/9377d6cb-e950-40bf-a232-f9646513fdc7-kube-api-access-77wmr\") pod \"payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22\" (UID: \"9377d6cb-e950-40bf-a232-f9646513fdc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"
Sep 12 01:08:23.496660 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.496556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9377d6cb-e950-40bf-a232-f9646513fdc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22\" (UID: \"9377d6cb-e950-40bf-a232-f9646513fdc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"
Sep 12 01:08:23.496660 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.496623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4pd\" (UniqueName: \"kubernetes.io/projected/10bd6015-9b3b-45ca-9731-aa5fce0b5f9a-kube-api-access-ms4pd\") pod \"payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m\" (UID: \"10bd6015-9b3b-45ca-9731-aa5fce0b5f9a\") " pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"
Sep 12 01:08:23.496660 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.496652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10bd6015-9b3b-45ca-9731-aa5fce0b5f9a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m\" (UID: \"10bd6015-9b3b-45ca-9731-aa5fce0b5f9a\") " pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"
Sep 12 01:08:23.496930 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.496697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77wmr\" (UniqueName: \"kubernetes.io/projected/9377d6cb-e950-40bf-a232-f9646513fdc7-kube-api-access-77wmr\") pod \"payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22\" (UID: \"9377d6cb-e950-40bf-a232-f9646513fdc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"
Sep 12 01:08:23.497291 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.497272    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9377d6cb-e950-40bf-a232-f9646513fdc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22\" (UID: \"9377d6cb-e950-40bf-a232-f9646513fdc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"
Sep 12 01:08:23.497336 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.497308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10bd6015-9b3b-45ca-9731-aa5fce0b5f9a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m\" (UID: \"10bd6015-9b3b-45ca-9731-aa5fce0b5f9a\") " pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"
Sep 12 01:08:23.506198 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.506166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms4pd\" (UniqueName: \"kubernetes.io/projected/10bd6015-9b3b-45ca-9731-aa5fce0b5f9a-kube-api-access-ms4pd\") pod \"payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m\" (UID: \"10bd6015-9b3b-45ca-9731-aa5fce0b5f9a\") " pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"
Sep 12 01:08:23.506320 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.506273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77wmr\" (UniqueName: \"kubernetes.io/projected/9377d6cb-e950-40bf-a232-f9646513fdc7-kube-api-access-77wmr\") pod \"payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22\" (UID: \"9377d6cb-e950-40bf-a232-f9646513fdc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"
Sep 12 01:08:23.535904 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.535876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"
Sep 12 01:08:23.576945 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.576317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"
Sep 12 01:08:23.824544 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.824513    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"]
Sep 12 01:08:23.826309 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:08:23.826283    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10bd6015_9b3b_45ca_9731_aa5fce0b5f9a.slice/crio-2d7291bb1b1133546d9f78d4407e8abd10aead20fab3aa4179ca7f568bfacd45 WatchSource:0}: Error finding container 2d7291bb1b1133546d9f78d4407e8abd10aead20fab3aa4179ca7f568bfacd45: Status 404 returned error can't find the container with id 2d7291bb1b1133546d9f78d4407e8abd10aead20fab3aa4179ca7f568bfacd45
Sep 12 01:08:23.861501 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:23.861481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"]
Sep 12 01:08:23.864780 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:08:23.864759    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9377d6cb_e950_40bf_a232_f9646513fdc7.slice/crio-e169920792bb5d0dbb7855a67ed70253d66430d18990a93f0c5a180c21e86752 WatchSource:0}: Error finding container e169920792bb5d0dbb7855a67ed70253d66430d18990a93f0c5a180c21e86752: Status 404 returned error can't find the container with id e169920792bb5d0dbb7855a67ed70253d66430d18990a93f0c5a180c21e86752
Sep 12 01:08:24.784835 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:24.784799    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22" event={"ID":"9377d6cb-e950-40bf-a232-f9646513fdc7","Type":"ContainerStarted","Data":"1bfda8d0d76851b8eff5ff50b859ccc2ee343fa781a8871c0822d8b2c80b2b23"}
Sep 12 01:08:24.785258 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:24.784856    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22" event={"ID":"9377d6cb-e950-40bf-a232-f9646513fdc7","Type":"ContainerStarted","Data":"e169920792bb5d0dbb7855a67ed70253d66430d18990a93f0c5a180c21e86752"}
Sep 12 01:08:24.785258 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:24.784883    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"
Sep 12 01:08:24.786388 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:24.786368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m" event={"ID":"10bd6015-9b3b-45ca-9731-aa5fce0b5f9a","Type":"ContainerStarted","Data":"758a0864b80ba439f6e5c8f318d19e19558464d4bba7042b79e861c34813e301"}
Sep 12 01:08:24.786487 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:24.786406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m" event={"ID":"10bd6015-9b3b-45ca-9731-aa5fce0b5f9a","Type":"ContainerStarted","Data":"2d7291bb1b1133546d9f78d4407e8abd10aead20fab3aa4179ca7f568bfacd45"}
Sep 12 01:08:24.786487 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:24.786425    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"
Sep 12 01:08:24.802311 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:24.802265    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22" podStartSLOduration=1.80225049 podStartE2EDuration="1.80225049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:08:24.800416611 +0000 UTC m=+1529.719708400" watchObservedRunningTime="2026-09-12 01:08:24.80225049 +0000 UTC m=+1529.721542277"
Sep 12 01:08:24.817592 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:24.817548    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m" podStartSLOduration=1.817536206 podStartE2EDuration="1.817536206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:08:24.815987977 +0000 UTC m=+1529.735279777" watchObservedRunningTime="2026-09-12 01:08:24.817536206 +0000 UTC m=+1529.736827992"
Sep 12 01:08:34.838144 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:34.838106    2579 generic.go:356] "Generic (PLEG): container finished" podID="1a4985c0-2f5f-4392-a8d5-a0eaaf913444" containerID="32eac081121d4e8724e6365b5402f453490a76341a062f6493fcd4d12c124215" exitCode=137
Sep 12 01:08:34.838550 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:34.838184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66" event={"ID":"1a4985c0-2f5f-4392-a8d5-a0eaaf913444","Type":"ContainerDied","Data":"32eac081121d4e8724e6365b5402f453490a76341a062f6493fcd4d12c124215"}
Sep 12 01:08:34.838550 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:34.838488    2579 scope.go:117] "RemoveContainer" containerID="32eac081121d4e8724e6365b5402f453490a76341a062f6493fcd4d12c124215"
Sep 12 01:08:35.793681 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:35.793647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"
Sep 12 01:08:35.793876 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:35.793716    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"
Sep 12 01:08:35.847656 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:35.847617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66" event={"ID":"1a4985c0-2f5f-4392-a8d5-a0eaaf913444","Type":"ContainerStarted","Data":"66bd9f00bb396479c3b579a1f63f5aca349e49cf7e1ea31f371113819eb0024f"}
Sep 12 01:08:35.848111 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:35.847830    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"
Sep 12 01:08:46.855297 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:46.855263    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"
Sep 12 01:08:55.762196 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:08:55.762163    2579 scope.go:117] "RemoveContainer" containerID="25a38b85026dd71ccef0b0bc4127013f6c21d1f584565ec83a322f760bd77aaa"
Sep 12 01:09:04.987999 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:04.987966    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"]
Sep 12 01:09:04.988579 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:04.988164    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j" podUID="a90ffb57-a008-4267-b9c9-3609b0b8c8ec" containerName="payload-processing" containerID="cri-o://ae7fd6e1876c223a4baad22535fcf8481e4034fa3f2080ac7c22407234c7574d" gracePeriod=30
Sep 12 01:09:04.994805 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:04.994778    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"]
Sep 12 01:09:04.995065 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:04.995035    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm" podUID="a7ed3812-2d25-45a8-b749-9610e5ada035" containerName="payload-pre-processing" containerID="cri-o://7776c433505b7cbacb3ddeb6dd27271149b2b55c0141fb9faea0739d3bc1f7f8" gracePeriod=30
Sep 12 01:09:05.250498 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:05.250427    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"
Sep 12 01:09:05.292930 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:05.292904    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"
Sep 12 01:09:05.362882 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:05.362847    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szkrx\" (UniqueName: \"kubernetes.io/projected/a90ffb57-a008-4267-b9c9-3609b0b8c8ec-kube-api-access-szkrx\") pod \"a90ffb57-a008-4267-b9c9-3609b0b8c8ec\" (UID: \"a90ffb57-a008-4267-b9c9-3609b0b8c8ec\") "
Sep 12 01:09:05.363057 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:05.362962    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90ffb57-a008-4267-b9c9-3609b0b8c8ec-plugins-config-volume\") pod \"a90ffb57-a008-4267-b9c9-3609b0b8c8ec\" (UID: \"a90ffb57-a008-4267-b9c9-3609b0b8c8ec\") "
Sep 12 01:09:05.363347 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:05.363314    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a90ffb57-a008-4267-b9c9-3609b0b8c8ec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a90ffb57-a008-4267-b9c9-3609b0b8c8ec" (UID: "a90ffb57-a008-4267-b9c9-3609b0b8c8ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:05.365283 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:05.365256    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90ffb57-a008-4267-b9c9-3609b0b8c8ec-kube-api-access-szkrx" (OuterVolumeSpecName: "kube-api-access-szkrx") pod "a90ffb57-a008-4267-b9c9-3609b0b8c8ec" (UID: "a90ffb57-a008-4267-b9c9-3609b0b8c8ec"). InnerVolumeSpecName "kube-api-access-szkrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:09:05.463914 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:05.463882    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mffb\" (UniqueName: \"kubernetes.io/projected/a7ed3812-2d25-45a8-b749-9610e5ada035-kube-api-access-8mffb\") pod \"a7ed3812-2d25-45a8-b749-9610e5ada035\" (UID: \"a7ed3812-2d25-45a8-b749-9610e5ada035\") "
Sep 12 01:09:05.464080 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:05.464002    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ed3812-2d25-45a8-b749-9610e5ada035-plugins-config-volume\") pod \"a7ed3812-2d25-45a8-b749-9610e5ada035\" (UID: \"a7ed3812-2d25-45a8-b749-9610e5ada035\") "
Sep 12 01:09:05.464200 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:05.464187    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a90ffb57-a008-4267-b9c9-3609b0b8c8ec-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:05.464268 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:05.464201    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szkrx\" (UniqueName: \"kubernetes.io/projected/a90ffb57-a008-4267-b9c9-3609b0b8c8ec-kube-api-access-szkrx\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:05.464409 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:05.464381    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ed3812-2d25-45a8-b749-9610e5ada035-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7ed3812-2d25-45a8-b749-9610e5ada035" (UID: "a7ed3812-2d25-45a8-b749-9610e5ada035"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:05.466087 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:05.466066    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ed3812-2d25-45a8-b749-9610e5ada035-kube-api-access-8mffb" (OuterVolumeSpecName: "kube-api-access-8mffb") pod "a7ed3812-2d25-45a8-b749-9610e5ada035" (UID: "a7ed3812-2d25-45a8-b749-9610e5ada035"). InnerVolumeSpecName "kube-api-access-8mffb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:09:05.564896 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:05.564802    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7ed3812-2d25-45a8-b749-9610e5ada035-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:05.564896 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:05.564840    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mffb\" (UniqueName: \"kubernetes.io/projected/a7ed3812-2d25-45a8-b749-9610e5ada035-kube-api-access-8mffb\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:06.022303 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.022268    2579 generic.go:356] "Generic (PLEG): container finished" podID="a90ffb57-a008-4267-b9c9-3609b0b8c8ec" containerID="ae7fd6e1876c223a4baad22535fcf8481e4034fa3f2080ac7c22407234c7574d" exitCode=0
Sep 12 01:09:06.022760 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.022326    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"
Sep 12 01:09:06.022760 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.022360    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j" event={"ID":"a90ffb57-a008-4267-b9c9-3609b0b8c8ec","Type":"ContainerDied","Data":"ae7fd6e1876c223a4baad22535fcf8481e4034fa3f2080ac7c22407234c7574d"}
Sep 12 01:09:06.022760 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.022439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j" event={"ID":"a90ffb57-a008-4267-b9c9-3609b0b8c8ec","Type":"ContainerDied","Data":"4549d9a02f5e2cb19c1164286f6ed05950d591924811515c325f5c7f71a1731b"}
Sep 12 01:09:06.022760 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.022494    2579 scope.go:117] "RemoveContainer" containerID="ae7fd6e1876c223a4baad22535fcf8481e4034fa3f2080ac7c22407234c7574d"
Sep 12 01:09:06.025131 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.024740    2579 generic.go:356] "Generic (PLEG): container finished" podID="a7ed3812-2d25-45a8-b749-9610e5ada035" containerID="7776c433505b7cbacb3ddeb6dd27271149b2b55c0141fb9faea0739d3bc1f7f8" exitCode=0
Sep 12 01:09:06.025131 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.024781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm" event={"ID":"a7ed3812-2d25-45a8-b749-9610e5ada035","Type":"ContainerDied","Data":"7776c433505b7cbacb3ddeb6dd27271149b2b55c0141fb9faea0739d3bc1f7f8"}
Sep 12 01:09:06.025131 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.024822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm" event={"ID":"a7ed3812-2d25-45a8-b749-9610e5ada035","Type":"ContainerDied","Data":"c05a414f569ff596672667a7dd7511a51e8c84aefe5efb62bb99965ccdf277f1"}
Sep 12 01:09:06.025131 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.024884    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"
Sep 12 01:09:06.035091 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.035073    2579 scope.go:117] "RemoveContainer" containerID="ae7fd6e1876c223a4baad22535fcf8481e4034fa3f2080ac7c22407234c7574d"
Sep 12 01:09:06.035464 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:09:06.035437    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae7fd6e1876c223a4baad22535fcf8481e4034fa3f2080ac7c22407234c7574d\": container with ID starting with ae7fd6e1876c223a4baad22535fcf8481e4034fa3f2080ac7c22407234c7574d not found: ID does not exist" containerID="ae7fd6e1876c223a4baad22535fcf8481e4034fa3f2080ac7c22407234c7574d"
Sep 12 01:09:06.035574 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.035476    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae7fd6e1876c223a4baad22535fcf8481e4034fa3f2080ac7c22407234c7574d"} err="failed to get container status \"ae7fd6e1876c223a4baad22535fcf8481e4034fa3f2080ac7c22407234c7574d\": rpc error: code = NotFound desc = could not find container \"ae7fd6e1876c223a4baad22535fcf8481e4034fa3f2080ac7c22407234c7574d\": container with ID starting with ae7fd6e1876c223a4baad22535fcf8481e4034fa3f2080ac7c22407234c7574d not found: ID does not exist"
Sep 12 01:09:06.035574 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.035500    2579 scope.go:117] "RemoveContainer" containerID="7776c433505b7cbacb3ddeb6dd27271149b2b55c0141fb9faea0739d3bc1f7f8"
Sep 12 01:09:06.045076 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.045019    2579 scope.go:117] "RemoveContainer" containerID="7776c433505b7cbacb3ddeb6dd27271149b2b55c0141fb9faea0739d3bc1f7f8"
Sep 12 01:09:06.045348 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:09:06.045331    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7776c433505b7cbacb3ddeb6dd27271149b2b55c0141fb9faea0739d3bc1f7f8\": container with ID starting with 7776c433505b7cbacb3ddeb6dd27271149b2b55c0141fb9faea0739d3bc1f7f8 not found: ID does not exist" containerID="7776c433505b7cbacb3ddeb6dd27271149b2b55c0141fb9faea0739d3bc1f7f8"
Sep 12 01:09:06.045416 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.045366    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7776c433505b7cbacb3ddeb6dd27271149b2b55c0141fb9faea0739d3bc1f7f8"} err="failed to get container status \"7776c433505b7cbacb3ddeb6dd27271149b2b55c0141fb9faea0739d3bc1f7f8\": rpc error: code = NotFound desc = could not find container \"7776c433505b7cbacb3ddeb6dd27271149b2b55c0141fb9faea0739d3bc1f7f8\": container with ID starting with 7776c433505b7cbacb3ddeb6dd27271149b2b55c0141fb9faea0739d3bc1f7f8 not found: ID does not exist"
Sep 12 01:09:06.046348 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.046322    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"]
Sep 12 01:09:06.050025 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.050004    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-033ca6-5dfc9cbb79-6q44j"]
Sep 12 01:09:06.059350 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.059324    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"]
Sep 12 01:09:06.064587 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:06.064563    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-033ca6-7cdcf6f6c8-crdlm"]
Sep 12 01:09:07.591827 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:07.591795    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7ed3812-2d25-45a8-b749-9610e5ada035" path="/var/lib/kubelet/pods/a7ed3812-2d25-45a8-b749-9610e5ada035/volumes"
Sep 12 01:09:07.592196 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:07.592131    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a90ffb57-a008-4267-b9c9-3609b0b8c8ec" path="/var/lib/kubelet/pods/a90ffb57-a008-4267-b9c9-3609b0b8c8ec/volumes"
Sep 12 01:09:12.681705 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:12.681660    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49980f-s8t89"]
Sep 12 01:09:12.682087 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:12.682074    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a90ffb57-a008-4267-b9c9-3609b0b8c8ec" containerName="payload-processing"
Sep 12 01:09:12.682087 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:12.682088    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a90ffb57-a008-4267-b9c9-3609b0b8c8ec" containerName="payload-processing"
Sep 12 01:09:12.682175 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:12.682106    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ed3812-2d25-45a8-b749-9610e5ada035" containerName="payload-pre-processing"
Sep 12 01:09:12.682175 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:12.682112    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ed3812-2d25-45a8-b749-9610e5ada035" containerName="payload-pre-processing"
Sep 12 01:09:12.682242 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:12.682189    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a90ffb57-a008-4267-b9c9-3609b0b8c8ec" containerName="payload-processing"
Sep 12 01:09:12.682242 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:12.682198    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ed3812-2d25-45a8-b749-9610e5ada035" containerName="payload-pre-processing"
Sep 12 01:09:12.686808 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:12.686791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49980f-s8t89"
Sep 12 01:09:12.689497 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:12.689474    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:09:12.689638 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:12.689544    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2rn9k\""
Sep 12 01:09:12.692838 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:12.692811    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49980f-s8t89"]
Sep 12 01:09:12.833047 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:12.833011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0da4d15-4289-4bf9-afe5-a6d26da81386-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49980f-s8t89\" (UID: \"d0da4d15-4289-4bf9-afe5-a6d26da81386\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49980f-s8t89"
Sep 12 01:09:12.934506 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:12.934418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0da4d15-4289-4bf9-afe5-a6d26da81386-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49980f-s8t89\" (UID: \"d0da4d15-4289-4bf9-afe5-a6d26da81386\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49980f-s8t89"
Sep 12 01:09:12.935080 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:12.935055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0da4d15-4289-4bf9-afe5-a6d26da81386-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-49980f-s8t89\" (UID: \"d0da4d15-4289-4bf9-afe5-a6d26da81386\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49980f-s8t89"
Sep 12 01:09:12.998668 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:12.998632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49980f-s8t89"
Sep 12 01:09:13.270707 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:13.270681    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49980f-s8t89"]
Sep 12 01:09:13.273187 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:09:13.273159    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0da4d15_4289_4bf9_afe5_a6d26da81386.slice/crio-6ef88011429b531f587ae9d4b48abc23e5529f184d020019fb01ecb5cbcdf2f2 WatchSource:0}: Error finding container 6ef88011429b531f587ae9d4b48abc23e5529f184d020019fb01ecb5cbcdf2f2: Status 404 returned error can't find the container with id 6ef88011429b531f587ae9d4b48abc23e5529f184d020019fb01ecb5cbcdf2f2
Sep 12 01:09:14.071527 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:14.071494    2579 generic.go:356] "Generic (PLEG): container finished" podID="d0da4d15-4289-4bf9-afe5-a6d26da81386" containerID="9ebc853fac6407c5393d5c489525a7749ce2e00502580edb58254d64a29f65da" exitCode=0
Sep 12 01:09:14.071888 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:14.071577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49980f-s8t89" event={"ID":"d0da4d15-4289-4bf9-afe5-a6d26da81386","Type":"ContainerDied","Data":"9ebc853fac6407c5393d5c489525a7749ce2e00502580edb58254d64a29f65da"}
Sep 12 01:09:14.071888 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:14.071649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49980f-s8t89" event={"ID":"d0da4d15-4289-4bf9-afe5-a6d26da81386","Type":"ContainerStarted","Data":"6ef88011429b531f587ae9d4b48abc23e5529f184d020019fb01ecb5cbcdf2f2"}
Sep 12 01:09:15.203983 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:15.203960    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49980f-s8t89"
Sep 12 01:09:15.255568 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:15.255062    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0da4d15-4289-4bf9-afe5-a6d26da81386-maas-api-service-ca\") pod \"d0da4d15-4289-4bf9-afe5-a6d26da81386\" (UID: \"d0da4d15-4289-4bf9-afe5-a6d26da81386\") "
Sep 12 01:09:15.255979 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:15.255933    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0da4d15-4289-4bf9-afe5-a6d26da81386-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0da4d15-4289-4bf9-afe5-a6d26da81386" (UID: "d0da4d15-4289-4bf9-afe5-a6d26da81386"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:15.355955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:15.355873    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0da4d15-4289-4bf9-afe5-a6d26da81386-maas-api-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:16.084128 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:16.084084    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49980f-s8t89"
Sep 12 01:09:16.084327 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:16.084088    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49980f-s8t89" event={"ID":"d0da4d15-4289-4bf9-afe5-a6d26da81386","Type":"ContainerDied","Data":"6ef88011429b531f587ae9d4b48abc23e5529f184d020019fb01ecb5cbcdf2f2"}
Sep 12 01:09:16.084327 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:16.084257    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ef88011429b531f587ae9d4b48abc23e5529f184d020019fb01ecb5cbcdf2f2"
Sep 12 01:09:28.479870 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.479834    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"]
Sep 12 01:09:28.480466 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.480446    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"]
Sep 12 01:09:28.480974 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.480918    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22" podUID="9377d6cb-e950-40bf-a232-f9646513fdc7" containerName="payload-pre-processing" containerID="cri-o://1bfda8d0d76851b8eff5ff50b859ccc2ee343fa781a8871c0822d8b2c80b2b23" gracePeriod=30
Sep 12 01:09:28.481560 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.481510    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m" podUID="10bd6015-9b3b-45ca-9731-aa5fce0b5f9a" containerName="payload-processing" containerID="cri-o://758a0864b80ba439f6e5c8f318d19e19558464d4bba7042b79e861c34813e301" gracePeriod=30
Sep 12 01:09:28.631995 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.631949    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49980f-s8t89"]
Sep 12 01:09:28.635282 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.635250    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-49980f-s8t89"]
Sep 12 01:09:28.763394 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.763367    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"
Sep 12 01:09:28.780579 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.780556    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"
Sep 12 01:09:28.884166 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.884130    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms4pd\" (UniqueName: \"kubernetes.io/projected/10bd6015-9b3b-45ca-9731-aa5fce0b5f9a-kube-api-access-ms4pd\") pod \"10bd6015-9b3b-45ca-9731-aa5fce0b5f9a\" (UID: \"10bd6015-9b3b-45ca-9731-aa5fce0b5f9a\") "
Sep 12 01:09:28.884166 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.884169    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10bd6015-9b3b-45ca-9731-aa5fce0b5f9a-plugins-config-volume\") pod \"10bd6015-9b3b-45ca-9731-aa5fce0b5f9a\" (UID: \"10bd6015-9b3b-45ca-9731-aa5fce0b5f9a\") "
Sep 12 01:09:28.884342 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.884193    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77wmr\" (UniqueName: \"kubernetes.io/projected/9377d6cb-e950-40bf-a232-f9646513fdc7-kube-api-access-77wmr\") pod \"9377d6cb-e950-40bf-a232-f9646513fdc7\" (UID: \"9377d6cb-e950-40bf-a232-f9646513fdc7\") "
Sep 12 01:09:28.884342 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.884229    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9377d6cb-e950-40bf-a232-f9646513fdc7-plugins-config-volume\") pod \"9377d6cb-e950-40bf-a232-f9646513fdc7\" (UID: \"9377d6cb-e950-40bf-a232-f9646513fdc7\") "
Sep 12 01:09:28.884644 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.884581    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10bd6015-9b3b-45ca-9731-aa5fce0b5f9a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "10bd6015-9b3b-45ca-9731-aa5fce0b5f9a" (UID: "10bd6015-9b3b-45ca-9731-aa5fce0b5f9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:28.884766 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.884652    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9377d6cb-e950-40bf-a232-f9646513fdc7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9377d6cb-e950-40bf-a232-f9646513fdc7" (UID: "9377d6cb-e950-40bf-a232-f9646513fdc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:28.886312 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.886290    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10bd6015-9b3b-45ca-9731-aa5fce0b5f9a-kube-api-access-ms4pd" (OuterVolumeSpecName: "kube-api-access-ms4pd") pod "10bd6015-9b3b-45ca-9731-aa5fce0b5f9a" (UID: "10bd6015-9b3b-45ca-9731-aa5fce0b5f9a"). InnerVolumeSpecName "kube-api-access-ms4pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:09:28.886384 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.886306    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9377d6cb-e950-40bf-a232-f9646513fdc7-kube-api-access-77wmr" (OuterVolumeSpecName: "kube-api-access-77wmr") pod "9377d6cb-e950-40bf-a232-f9646513fdc7" (UID: "9377d6cb-e950-40bf-a232-f9646513fdc7"). InnerVolumeSpecName "kube-api-access-77wmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:09:28.985083 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.985050    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms4pd\" (UniqueName: \"kubernetes.io/projected/10bd6015-9b3b-45ca-9731-aa5fce0b5f9a-kube-api-access-ms4pd\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:28.985083 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.985076    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10bd6015-9b3b-45ca-9731-aa5fce0b5f9a-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:28.985083 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.985086    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77wmr\" (UniqueName: \"kubernetes.io/projected/9377d6cb-e950-40bf-a232-f9646513fdc7-kube-api-access-77wmr\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:28.985083 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:28.985094    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9377d6cb-e950-40bf-a232-f9646513fdc7-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:29.159436 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.155392    2579 generic.go:356] "Generic (PLEG): container finished" podID="9377d6cb-e950-40bf-a232-f9646513fdc7" containerID="1bfda8d0d76851b8eff5ff50b859ccc2ee343fa781a8871c0822d8b2c80b2b23" exitCode=0
Sep 12 01:09:29.159436 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.155576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22" event={"ID":"9377d6cb-e950-40bf-a232-f9646513fdc7","Type":"ContainerDied","Data":"1bfda8d0d76851b8eff5ff50b859ccc2ee343fa781a8871c0822d8b2c80b2b23"}
Sep 12 01:09:29.159436 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.155663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22" event={"ID":"9377d6cb-e950-40bf-a232-f9646513fdc7","Type":"ContainerDied","Data":"e169920792bb5d0dbb7855a67ed70253d66430d18990a93f0c5a180c21e86752"}
Sep 12 01:09:29.159436 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.155700    2579 scope.go:117] "RemoveContainer" containerID="1bfda8d0d76851b8eff5ff50b859ccc2ee343fa781a8871c0822d8b2c80b2b23"
Sep 12 01:09:29.159436 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.155878    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"
Sep 12 01:09:29.161152 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.161122    2579 generic.go:356] "Generic (PLEG): container finished" podID="10bd6015-9b3b-45ca-9731-aa5fce0b5f9a" containerID="758a0864b80ba439f6e5c8f318d19e19558464d4bba7042b79e861c34813e301" exitCode=0
Sep 12 01:09:29.161270 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.161215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m" event={"ID":"10bd6015-9b3b-45ca-9731-aa5fce0b5f9a","Type":"ContainerDied","Data":"758a0864b80ba439f6e5c8f318d19e19558464d4bba7042b79e861c34813e301"}
Sep 12 01:09:29.161270 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.161231    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"
Sep 12 01:09:29.161270 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.161261    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m" event={"ID":"10bd6015-9b3b-45ca-9731-aa5fce0b5f9a","Type":"ContainerDied","Data":"2d7291bb1b1133546d9f78d4407e8abd10aead20fab3aa4179ca7f568bfacd45"}
Sep 12 01:09:29.169307 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.169278    2579 scope.go:117] "RemoveContainer" containerID="1bfda8d0d76851b8eff5ff50b859ccc2ee343fa781a8871c0822d8b2c80b2b23"
Sep 12 01:09:29.169574 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:09:29.169553    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfda8d0d76851b8eff5ff50b859ccc2ee343fa781a8871c0822d8b2c80b2b23\": container with ID starting with 1bfda8d0d76851b8eff5ff50b859ccc2ee343fa781a8871c0822d8b2c80b2b23 not found: ID does not exist" containerID="1bfda8d0d76851b8eff5ff50b859ccc2ee343fa781a8871c0822d8b2c80b2b23"
Sep 12 01:09:29.169677 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.169581    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfda8d0d76851b8eff5ff50b859ccc2ee343fa781a8871c0822d8b2c80b2b23"} err="failed to get container status \"1bfda8d0d76851b8eff5ff50b859ccc2ee343fa781a8871c0822d8b2c80b2b23\": rpc error: code = NotFound desc = could not find container \"1bfda8d0d76851b8eff5ff50b859ccc2ee343fa781a8871c0822d8b2c80b2b23\": container with ID starting with 1bfda8d0d76851b8eff5ff50b859ccc2ee343fa781a8871c0822d8b2c80b2b23 not found: ID does not exist"
Sep 12 01:09:29.169677 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.169624    2579 scope.go:117] "RemoveContainer" containerID="758a0864b80ba439f6e5c8f318d19e19558464d4bba7042b79e861c34813e301"
Sep 12 01:09:29.177590 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.177572    2579 scope.go:117] "RemoveContainer" containerID="758a0864b80ba439f6e5c8f318d19e19558464d4bba7042b79e861c34813e301"
Sep 12 01:09:29.177863 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:09:29.177844    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758a0864b80ba439f6e5c8f318d19e19558464d4bba7042b79e861c34813e301\": container with ID starting with 758a0864b80ba439f6e5c8f318d19e19558464d4bba7042b79e861c34813e301 not found: ID does not exist" containerID="758a0864b80ba439f6e5c8f318d19e19558464d4bba7042b79e861c34813e301"
Sep 12 01:09:29.177911 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.177870    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758a0864b80ba439f6e5c8f318d19e19558464d4bba7042b79e861c34813e301"} err="failed to get container status \"758a0864b80ba439f6e5c8f318d19e19558464d4bba7042b79e861c34813e301\": rpc error: code = NotFound desc = could not find container \"758a0864b80ba439f6e5c8f318d19e19558464d4bba7042b79e861c34813e301\": container with ID starting with 758a0864b80ba439f6e5c8f318d19e19558464d4bba7042b79e861c34813e301 not found: ID does not exist"
Sep 12 01:09:29.182096 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.182074    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"]
Sep 12 01:09:29.187282 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.187254    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49980f-765dd7fd78-nhv22"]
Sep 12 01:09:29.198662 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.198641    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"]
Sep 12 01:09:29.201139 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.201117    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-49980f-85c77cf6cc-fpn7m"]
Sep 12 01:09:29.403500 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.403459    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk"]
Sep 12 01:09:29.403930 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.403915    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0da4d15-4289-4bf9-afe5-a6d26da81386" containerName="revoke-keys"
Sep 12 01:09:29.403999 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.403932    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0da4d15-4289-4bf9-afe5-a6d26da81386" containerName="revoke-keys"
Sep 12 01:09:29.403999 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.403948    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10bd6015-9b3b-45ca-9731-aa5fce0b5f9a" containerName="payload-processing"
Sep 12 01:09:29.403999 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.403956    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="10bd6015-9b3b-45ca-9731-aa5fce0b5f9a" containerName="payload-processing"
Sep 12 01:09:29.403999 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.403980    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9377d6cb-e950-40bf-a232-f9646513fdc7" containerName="payload-pre-processing"
Sep 12 01:09:29.403999 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.403990    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9377d6cb-e950-40bf-a232-f9646513fdc7" containerName="payload-pre-processing"
Sep 12 01:09:29.404238 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.404071    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="10bd6015-9b3b-45ca-9731-aa5fce0b5f9a" containerName="payload-processing"
Sep 12 01:09:29.404238 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.404082    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9377d6cb-e950-40bf-a232-f9646513fdc7" containerName="payload-pre-processing"
Sep 12 01:09:29.404238 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.404095    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0da4d15-4289-4bf9-afe5-a6d26da81386" containerName="revoke-keys"
Sep 12 01:09:29.407551 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.407533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk"
Sep 12 01:09:29.410214 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.410164    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:09:29.410311 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.410268    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2rn9k\""
Sep 12 01:09:29.417383 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.417357    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk"]
Sep 12 01:09:29.490298 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.490269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40dc4b89-9cdd-4c00-bcaf-bb2950da67c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk\" (UID: \"40dc4b89-9cdd-4c00-bcaf-bb2950da67c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk"
Sep 12 01:09:29.591519 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.591492    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10bd6015-9b3b-45ca-9731-aa5fce0b5f9a" path="/var/lib/kubelet/pods/10bd6015-9b3b-45ca-9731-aa5fce0b5f9a/volumes"
Sep 12 01:09:29.591666 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.591651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40dc4b89-9cdd-4c00-bcaf-bb2950da67c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk\" (UID: \"40dc4b89-9cdd-4c00-bcaf-bb2950da67c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk"
Sep 12 01:09:29.591894 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.591880    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9377d6cb-e950-40bf-a232-f9646513fdc7" path="/var/lib/kubelet/pods/9377d6cb-e950-40bf-a232-f9646513fdc7/volumes"
Sep 12 01:09:29.592204 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.592187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40dc4b89-9cdd-4c00-bcaf-bb2950da67c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk\" (UID: \"40dc4b89-9cdd-4c00-bcaf-bb2950da67c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk"
Sep 12 01:09:29.592242 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.592201    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0da4d15-4289-4bf9-afe5-a6d26da81386" path="/var/lib/kubelet/pods/d0da4d15-4289-4bf9-afe5-a6d26da81386/volumes"
Sep 12 01:09:29.719089 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.719054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk"
Sep 12 01:09:29.991570 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:29.991496    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk"]
Sep 12 01:09:29.994575 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:09:29.994546    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40dc4b89_9cdd_4c00_bcaf_bb2950da67c1.slice/crio-8a14e58d1bc6433617237abac9b885118db9d1ffd4cd84baa305d09aa8aafe6f WatchSource:0}: Error finding container 8a14e58d1bc6433617237abac9b885118db9d1ffd4cd84baa305d09aa8aafe6f: Status 404 returned error can't find the container with id 8a14e58d1bc6433617237abac9b885118db9d1ffd4cd84baa305d09aa8aafe6f
Sep 12 01:09:30.168091 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:30.168059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk" event={"ID":"40dc4b89-9cdd-4c00-bcaf-bb2950da67c1","Type":"ContainerStarted","Data":"8a14e58d1bc6433617237abac9b885118db9d1ffd4cd84baa305d09aa8aafe6f"}
Sep 12 01:09:31.175086 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:31.175051    2579 generic.go:356] "Generic (PLEG): container finished" podID="40dc4b89-9cdd-4c00-bcaf-bb2950da67c1" containerID="07736b658343eb398327b6bd18cc6e6eadeec996dd7dedaf279eb955916d7ceb" exitCode=0
Sep 12 01:09:31.175445 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:31.175132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk" event={"ID":"40dc4b89-9cdd-4c00-bcaf-bb2950da67c1","Type":"ContainerDied","Data":"07736b658343eb398327b6bd18cc6e6eadeec996dd7dedaf279eb955916d7ceb"}
Sep 12 01:09:32.345773 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:32.345749    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk"
Sep 12 01:09:32.421173 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:32.421144    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40dc4b89-9cdd-4c00-bcaf-bb2950da67c1-maas-api-service-ca\") pod \"40dc4b89-9cdd-4c00-bcaf-bb2950da67c1\" (UID: \"40dc4b89-9cdd-4c00-bcaf-bb2950da67c1\") "
Sep 12 01:09:32.421503 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:32.421478    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40dc4b89-9cdd-4c00-bcaf-bb2950da67c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "40dc4b89-9cdd-4c00-bcaf-bb2950da67c1" (UID: "40dc4b89-9cdd-4c00-bcaf-bb2950da67c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:32.522489 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:32.522415    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40dc4b89-9cdd-4c00-bcaf-bb2950da67c1-maas-api-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:33.190536 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:33.190504    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk"
Sep 12 01:09:33.190706 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:33.190545    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk" event={"ID":"40dc4b89-9cdd-4c00-bcaf-bb2950da67c1","Type":"ContainerDied","Data":"8a14e58d1bc6433617237abac9b885118db9d1ffd4cd84baa305d09aa8aafe6f"}
Sep 12 01:09:33.190706 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:33.190594    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a14e58d1bc6433617237abac9b885118db9d1ffd4cd84baa305d09aa8aafe6f"
Sep 12 01:09:35.877728 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:35.877692    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv"]
Sep 12 01:09:35.878108 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:35.878095    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40dc4b89-9cdd-4c00-bcaf-bb2950da67c1" containerName="revoke-keys"
Sep 12 01:09:35.878108 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:35.878110    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="40dc4b89-9cdd-4c00-bcaf-bb2950da67c1" containerName="revoke-keys"
Sep 12 01:09:35.878207 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:35.878195    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="40dc4b89-9cdd-4c00-bcaf-bb2950da67c1" containerName="revoke-keys"
Sep 12 01:09:35.881234 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:35.881216    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv"
Sep 12 01:09:35.883471 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:35.883452    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:09:35.883586 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:35.883480    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2rn9k\""
Sep 12 01:09:35.888757 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:35.888635    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv"]
Sep 12 01:09:35.954267 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:35.954232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7052f9d-760c-45c8-b324-eeca4f9a139c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv\" (UID: \"f7052f9d-760c-45c8-b324-eeca4f9a139c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv"
Sep 12 01:09:36.055738 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:36.055706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7052f9d-760c-45c8-b324-eeca4f9a139c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv\" (UID: \"f7052f9d-760c-45c8-b324-eeca4f9a139c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv"
Sep 12 01:09:36.056302 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:36.056285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7052f9d-760c-45c8-b324-eeca4f9a139c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv\" (UID: \"f7052f9d-760c-45c8-b324-eeca4f9a139c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv"
Sep 12 01:09:36.192338 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:36.192296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv"
Sep 12 01:09:36.466705 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:36.466678    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv"]
Sep 12 01:09:36.468432 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:09:36.468394    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7052f9d_760c_45c8_b324_eeca4f9a139c.slice/crio-a83d124b0a65ad17787117b1cb38f294c7e3c401dc3d70abeb62ef486807f054 WatchSource:0}: Error finding container a83d124b0a65ad17787117b1cb38f294c7e3c401dc3d70abeb62ef486807f054: Status 404 returned error can't find the container with id a83d124b0a65ad17787117b1cb38f294c7e3c401dc3d70abeb62ef486807f054
Sep 12 01:09:37.213496 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:37.213455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv" event={"ID":"f7052f9d-760c-45c8-b324-eeca4f9a139c","Type":"ContainerStarted","Data":"9c4cf38e0fc7293d99814d65aff22d10af1c0ba54e554fefb04eb02691aa8f3c"}
Sep 12 01:09:37.213888 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:37.213524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv" event={"ID":"f7052f9d-760c-45c8-b324-eeca4f9a139c","Type":"ContainerStarted","Data":"a83d124b0a65ad17787117b1cb38f294c7e3c401dc3d70abeb62ef486807f054"}
Sep 12 01:09:37.229842 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:37.229794    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv" podStartSLOduration=2.2297796500000002 podStartE2EDuration="2.22977965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:09:37.22854685 +0000 UTC m=+1602.147838638" watchObservedRunningTime="2026-09-12 01:09:37.22977965 +0000 UTC m=+1602.149071437"
Sep 12 01:09:38.219920 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:38.219885    2579 generic.go:356] "Generic (PLEG): container finished" podID="f7052f9d-760c-45c8-b324-eeca4f9a139c" containerID="9c4cf38e0fc7293d99814d65aff22d10af1c0ba54e554fefb04eb02691aa8f3c" exitCode=0
Sep 12 01:09:38.220305 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:38.219978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv" event={"ID":"f7052f9d-760c-45c8-b324-eeca4f9a139c","Type":"ContainerDied","Data":"9c4cf38e0fc7293d99814d65aff22d10af1c0ba54e554fefb04eb02691aa8f3c"}
Sep 12 01:09:39.372234 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:39.372211    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv"
Sep 12 01:09:39.488235 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:39.488146    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7052f9d-760c-45c8-b324-eeca4f9a139c-maas-api-service-ca\") pod \"f7052f9d-760c-45c8-b324-eeca4f9a139c\" (UID: \"f7052f9d-760c-45c8-b324-eeca4f9a139c\") "
Sep 12 01:09:39.488535 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:39.488515    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7052f9d-760c-45c8-b324-eeca4f9a139c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7052f9d-760c-45c8-b324-eeca4f9a139c" (UID: "f7052f9d-760c-45c8-b324-eeca4f9a139c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:39.588802 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:39.588771    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7052f9d-760c-45c8-b324-eeca4f9a139c-maas-api-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:40.232806 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:40.232775    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv" event={"ID":"f7052f9d-760c-45c8-b324-eeca4f9a139c","Type":"ContainerDied","Data":"a83d124b0a65ad17787117b1cb38f294c7e3c401dc3d70abeb62ef486807f054"}
Sep 12 01:09:40.232806 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:40.232800    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv"
Sep 12 01:09:40.232806 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:40.232823    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a83d124b0a65ad17787117b1cb38f294c7e3c401dc3d70abeb62ef486807f054"
Sep 12 01:09:45.108976 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.108860    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"]
Sep 12 01:09:45.109438 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.109102    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs" podUID="bdca3a0b-aea6-42f7-8550-4a0f4d23209b" containerName="payload-processing" containerID="cri-o://110284eadc9c9790603563ba76254ae1b4e4f8cb583fa98d7b390b1926d4000f" gracePeriod=30
Sep 12 01:09:45.119047 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.119012    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"]
Sep 12 01:09:45.119265 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.119240    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl" podUID="968de32b-1dea-422a-a1aa-dc4f78412fbb" containerName="payload-pre-processing" containerID="cri-o://165045b60efdd6f6e98b91583dd79945c39f5fa757d34ac3766fbfb7b2d48ddd" gracePeriod=30
Sep 12 01:09:45.261402 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.261366    2579 generic.go:356] "Generic (PLEG): container finished" podID="bdca3a0b-aea6-42f7-8550-4a0f4d23209b" containerID="110284eadc9c9790603563ba76254ae1b4e4f8cb583fa98d7b390b1926d4000f" exitCode=0
Sep 12 01:09:45.261574 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.261447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs" event={"ID":"bdca3a0b-aea6-42f7-8550-4a0f4d23209b","Type":"ContainerDied","Data":"110284eadc9c9790603563ba76254ae1b4e4f8cb583fa98d7b390b1926d4000f"}
Sep 12 01:09:45.263170 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.263144    2579 generic.go:356] "Generic (PLEG): container finished" podID="968de32b-1dea-422a-a1aa-dc4f78412fbb" containerID="165045b60efdd6f6e98b91583dd79945c39f5fa757d34ac3766fbfb7b2d48ddd" exitCode=0
Sep 12 01:09:45.263305 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.263211    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl" event={"ID":"968de32b-1dea-422a-a1aa-dc4f78412fbb","Type":"ContainerDied","Data":"165045b60efdd6f6e98b91583dd79945c39f5fa757d34ac3766fbfb7b2d48ddd"}
Sep 12 01:09:45.313157 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.313120    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk"]
Sep 12 01:09:45.316726 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.316697    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10ba6f-rj6gk"]
Sep 12 01:09:45.383957 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.383935    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"
Sep 12 01:09:45.411189 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.411163    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"
Sep 12 01:09:45.542236 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.542204    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r7q8\" (UniqueName: \"kubernetes.io/projected/bdca3a0b-aea6-42f7-8550-4a0f4d23209b-kube-api-access-2r7q8\") pod \"bdca3a0b-aea6-42f7-8550-4a0f4d23209b\" (UID: \"bdca3a0b-aea6-42f7-8550-4a0f4d23209b\") "
Sep 12 01:09:45.542404 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.542253    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnr6r\" (UniqueName: \"kubernetes.io/projected/968de32b-1dea-422a-a1aa-dc4f78412fbb-kube-api-access-dnr6r\") pod \"968de32b-1dea-422a-a1aa-dc4f78412fbb\" (UID: \"968de32b-1dea-422a-a1aa-dc4f78412fbb\") "
Sep 12 01:09:45.542404 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.542286    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdca3a0b-aea6-42f7-8550-4a0f4d23209b-plugins-config-volume\") pod \"bdca3a0b-aea6-42f7-8550-4a0f4d23209b\" (UID: \"bdca3a0b-aea6-42f7-8550-4a0f4d23209b\") "
Sep 12 01:09:45.542404 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.542390    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968de32b-1dea-422a-a1aa-dc4f78412fbb-plugins-config-volume\") pod \"968de32b-1dea-422a-a1aa-dc4f78412fbb\" (UID: \"968de32b-1dea-422a-a1aa-dc4f78412fbb\") "
Sep 12 01:09:45.542774 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.542747    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdca3a0b-aea6-42f7-8550-4a0f4d23209b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bdca3a0b-aea6-42f7-8550-4a0f4d23209b" (UID: "bdca3a0b-aea6-42f7-8550-4a0f4d23209b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:45.542880 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.542781    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/968de32b-1dea-422a-a1aa-dc4f78412fbb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "968de32b-1dea-422a-a1aa-dc4f78412fbb" (UID: "968de32b-1dea-422a-a1aa-dc4f78412fbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:45.544283 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.544262    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdca3a0b-aea6-42f7-8550-4a0f4d23209b-kube-api-access-2r7q8" (OuterVolumeSpecName: "kube-api-access-2r7q8") pod "bdca3a0b-aea6-42f7-8550-4a0f4d23209b" (UID: "bdca3a0b-aea6-42f7-8550-4a0f4d23209b"). InnerVolumeSpecName "kube-api-access-2r7q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:09:45.544325 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.544268    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968de32b-1dea-422a-a1aa-dc4f78412fbb-kube-api-access-dnr6r" (OuterVolumeSpecName: "kube-api-access-dnr6r") pod "968de32b-1dea-422a-a1aa-dc4f78412fbb" (UID: "968de32b-1dea-422a-a1aa-dc4f78412fbb"). InnerVolumeSpecName "kube-api-access-dnr6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:09:45.590535 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.590506    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40dc4b89-9cdd-4c00-bcaf-bb2950da67c1" path="/var/lib/kubelet/pods/40dc4b89-9cdd-4c00-bcaf-bb2950da67c1/volumes"
Sep 12 01:09:45.643538 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.643463    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968de32b-1dea-422a-a1aa-dc4f78412fbb-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:45.643538 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.643489    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r7q8\" (UniqueName: \"kubernetes.io/projected/bdca3a0b-aea6-42f7-8550-4a0f4d23209b-kube-api-access-2r7q8\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:45.643538 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.643498    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnr6r\" (UniqueName: \"kubernetes.io/projected/968de32b-1dea-422a-a1aa-dc4f78412fbb-kube-api-access-dnr6r\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:45.643538 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:45.643508    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdca3a0b-aea6-42f7-8550-4a0f4d23209b-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:46.270411 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:46.270375    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"
Sep 12 01:09:46.270411 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:46.270391    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs" event={"ID":"bdca3a0b-aea6-42f7-8550-4a0f4d23209b","Type":"ContainerDied","Data":"a0afc2bd4078a967d6d6aee419d4016824bc76dd3077d98de570e6e2ce53e28f"}
Sep 12 01:09:46.270970 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:46.270451    2579 scope.go:117] "RemoveContainer" containerID="110284eadc9c9790603563ba76254ae1b4e4f8cb583fa98d7b390b1926d4000f"
Sep 12 01:09:46.271906 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:46.271892    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"
Sep 12 01:09:46.271984 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:46.271943    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl" event={"ID":"968de32b-1dea-422a-a1aa-dc4f78412fbb","Type":"ContainerDied","Data":"e0f6ef4a1490c387341392b16883666f28f6dda3d05ada66e2f170c469198e82"}
Sep 12 01:09:46.281955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:46.281934    2579 scope.go:117] "RemoveContainer" containerID="165045b60efdd6f6e98b91583dd79945c39f5fa757d34ac3766fbfb7b2d48ddd"
Sep 12 01:09:46.291423 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:46.291405    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"]
Sep 12 01:09:46.297707 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:46.296048    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10ba6f-7dd6468598-x4wtl"]
Sep 12 01:09:46.311646 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:46.311624    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"]
Sep 12 01:09:46.315635 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:46.315613    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10ba6f-d456458c-wfvzs"]
Sep 12 01:09:47.592402 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:47.592370    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="968de32b-1dea-422a-a1aa-dc4f78412fbb" path="/var/lib/kubelet/pods/968de32b-1dea-422a-a1aa-dc4f78412fbb/volumes"
Sep 12 01:09:47.592764 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:47.592742    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdca3a0b-aea6-42f7-8550-4a0f4d23209b" path="/var/lib/kubelet/pods/bdca3a0b-aea6-42f7-8550-4a0f4d23209b/volumes"
Sep 12 01:09:48.158662 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.158628    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"]
Sep 12 01:09:48.158869 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.158845    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879" podUID="e9aeea73-045e-4e21-b0ce-644cf60649ad" containerName="payload-processing" containerID="cri-o://b3b5cbfba5c325c933d0e192fe1bcc37916a01aa487dd62f814b57749df81956" gracePeriod=30
Sep 12 01:09:48.164796 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.164772    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"]
Sep 12 01:09:48.165196 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.165135    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq" podUID="dbcdc926-82c2-41d7-a231-81280d5d7864" containerName="payload-pre-processing" containerID="cri-o://4def96c38eca5d3e72bc34c5eafca1df505f85679f7d1ca63c9953b01bdb974d" gracePeriod=30
Sep 12 01:09:48.286676 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.286640    2579 generic.go:356] "Generic (PLEG): container finished" podID="dbcdc926-82c2-41d7-a231-81280d5d7864" containerID="4def96c38eca5d3e72bc34c5eafca1df505f85679f7d1ca63c9953b01bdb974d" exitCode=0
Sep 12 01:09:48.286799 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.286660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq" event={"ID":"dbcdc926-82c2-41d7-a231-81280d5d7864","Type":"ContainerDied","Data":"4def96c38eca5d3e72bc34c5eafca1df505f85679f7d1ca63c9953b01bdb974d"}
Sep 12 01:09:48.288617 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.288576    2579 generic.go:356] "Generic (PLEG): container finished" podID="e9aeea73-045e-4e21-b0ce-644cf60649ad" containerID="b3b5cbfba5c325c933d0e192fe1bcc37916a01aa487dd62f814b57749df81956" exitCode=0
Sep 12 01:09:48.288712 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.288665    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879" event={"ID":"e9aeea73-045e-4e21-b0ce-644cf60649ad","Type":"ContainerDied","Data":"b3b5cbfba5c325c933d0e192fe1bcc37916a01aa487dd62f814b57749df81956"}
Sep 12 01:09:48.351914 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.351881    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv"]
Sep 12 01:09:48.356816 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.356783    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e62c2-8mdhv"]
Sep 12 01:09:48.423799 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.423775    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"
Sep 12 01:09:48.456489 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.456468    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"
Sep 12 01:09:48.570352 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.570324    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4l4c\" (UniqueName: \"kubernetes.io/projected/dbcdc926-82c2-41d7-a231-81280d5d7864-kube-api-access-q4l4c\") pod \"dbcdc926-82c2-41d7-a231-81280d5d7864\" (UID: \"dbcdc926-82c2-41d7-a231-81280d5d7864\") "
Sep 12 01:09:48.570551 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.570362    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9aeea73-045e-4e21-b0ce-644cf60649ad-plugins-config-volume\") pod \"e9aeea73-045e-4e21-b0ce-644cf60649ad\" (UID: \"e9aeea73-045e-4e21-b0ce-644cf60649ad\") "
Sep 12 01:09:48.570551 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.570422    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stlht\" (UniqueName: \"kubernetes.io/projected/e9aeea73-045e-4e21-b0ce-644cf60649ad-kube-api-access-stlht\") pod \"e9aeea73-045e-4e21-b0ce-644cf60649ad\" (UID: \"e9aeea73-045e-4e21-b0ce-644cf60649ad\") "
Sep 12 01:09:48.570551 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.570459    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbcdc926-82c2-41d7-a231-81280d5d7864-plugins-config-volume\") pod \"dbcdc926-82c2-41d7-a231-81280d5d7864\" (UID: \"dbcdc926-82c2-41d7-a231-81280d5d7864\") "
Sep 12 01:09:48.570831 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.570803    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9aeea73-045e-4e21-b0ce-644cf60649ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9aeea73-045e-4e21-b0ce-644cf60649ad" (UID: "e9aeea73-045e-4e21-b0ce-644cf60649ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:48.570894 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.570838    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbcdc926-82c2-41d7-a231-81280d5d7864-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dbcdc926-82c2-41d7-a231-81280d5d7864" (UID: "dbcdc926-82c2-41d7-a231-81280d5d7864"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:48.572462 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.572441    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9aeea73-045e-4e21-b0ce-644cf60649ad-kube-api-access-stlht" (OuterVolumeSpecName: "kube-api-access-stlht") pod "e9aeea73-045e-4e21-b0ce-644cf60649ad" (UID: "e9aeea73-045e-4e21-b0ce-644cf60649ad"). InnerVolumeSpecName "kube-api-access-stlht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:09:48.572512 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.572471    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbcdc926-82c2-41d7-a231-81280d5d7864-kube-api-access-q4l4c" (OuterVolumeSpecName: "kube-api-access-q4l4c") pod "dbcdc926-82c2-41d7-a231-81280d5d7864" (UID: "dbcdc926-82c2-41d7-a231-81280d5d7864"). InnerVolumeSpecName "kube-api-access-q4l4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:09:48.671655 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.671621    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4l4c\" (UniqueName: \"kubernetes.io/projected/dbcdc926-82c2-41d7-a231-81280d5d7864-kube-api-access-q4l4c\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:48.671655 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.671648    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9aeea73-045e-4e21-b0ce-644cf60649ad-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:48.671655 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.671657    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stlht\" (UniqueName: \"kubernetes.io/projected/e9aeea73-045e-4e21-b0ce-644cf60649ad-kube-api-access-stlht\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:48.672068 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:48.671667    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbcdc926-82c2-41d7-a231-81280d5d7864-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:49.295074 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:49.295046    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"
Sep 12 01:09:49.295230 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:49.295076    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq" event={"ID":"dbcdc926-82c2-41d7-a231-81280d5d7864","Type":"ContainerDied","Data":"00cdc57e51942f415f59ce9ba49149de894d5a8b0455a6e96696664b0c6f4bd7"}
Sep 12 01:09:49.295230 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:49.295137    2579 scope.go:117] "RemoveContainer" containerID="4def96c38eca5d3e72bc34c5eafca1df505f85679f7d1ca63c9953b01bdb974d"
Sep 12 01:09:49.296539 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:49.296520    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"
Sep 12 01:09:49.296642 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:49.296523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879" event={"ID":"e9aeea73-045e-4e21-b0ce-644cf60649ad","Type":"ContainerDied","Data":"395f063817b2601473a4c33a6a357311bafefa3a918ec44756cefbf2d7235ae4"}
Sep 12 01:09:49.305995 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:49.305974    2579 scope.go:117] "RemoveContainer" containerID="b3b5cbfba5c325c933d0e192fe1bcc37916a01aa487dd62f814b57749df81956"
Sep 12 01:09:49.330556 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:49.330535    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"]
Sep 12 01:09:49.345027 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:49.345005    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8e62c2-7dc9c9d989-sq879"]
Sep 12 01:09:49.364770 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:49.364751    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"]
Sep 12 01:09:49.372013 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:49.371992    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e62c2-5459cb9c94-sn9vq"]
Sep 12 01:09:49.596013 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:49.592770    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbcdc926-82c2-41d7-a231-81280d5d7864" path="/var/lib/kubelet/pods/dbcdc926-82c2-41d7-a231-81280d5d7864/volumes"
Sep 12 01:09:49.596013 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:49.593319    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9aeea73-045e-4e21-b0ce-644cf60649ad" path="/var/lib/kubelet/pods/e9aeea73-045e-4e21-b0ce-644cf60649ad/volumes"
Sep 12 01:09:49.596013 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:49.593792    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7052f9d-760c-45c8-b324-eeca4f9a139c" path="/var/lib/kubelet/pods/f7052f9d-760c-45c8-b324-eeca4f9a139c/volumes"
Sep 12 01:09:55.214197 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.214155    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2"]
Sep 12 01:09:55.214760 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.214738    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbcdc926-82c2-41d7-a231-81280d5d7864" containerName="payload-pre-processing"
Sep 12 01:09:55.214812 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.214765    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbcdc926-82c2-41d7-a231-81280d5d7864" containerName="payload-pre-processing"
Sep 12 01:09:55.214812 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.214778    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="968de32b-1dea-422a-a1aa-dc4f78412fbb" containerName="payload-pre-processing"
Sep 12 01:09:55.214812 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.214787    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="968de32b-1dea-422a-a1aa-dc4f78412fbb" containerName="payload-pre-processing"
Sep 12 01:09:55.214812 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.214807    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdca3a0b-aea6-42f7-8550-4a0f4d23209b" containerName="payload-processing"
Sep 12 01:09:55.214935 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.214816    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdca3a0b-aea6-42f7-8550-4a0f4d23209b" containerName="payload-processing"
Sep 12 01:09:55.214935 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.214832    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9aeea73-045e-4e21-b0ce-644cf60649ad" containerName="payload-processing"
Sep 12 01:09:55.214935 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.214840    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9aeea73-045e-4e21-b0ce-644cf60649ad" containerName="payload-processing"
Sep 12 01:09:55.214935 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.214857    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7052f9d-760c-45c8-b324-eeca4f9a139c" containerName="revoke-keys"
Sep 12 01:09:55.214935 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.214866    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7052f9d-760c-45c8-b324-eeca4f9a139c" containerName="revoke-keys"
Sep 12 01:09:55.215086 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.214967    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbcdc926-82c2-41d7-a231-81280d5d7864" containerName="payload-pre-processing"
Sep 12 01:09:55.215086 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.214986    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7052f9d-760c-45c8-b324-eeca4f9a139c" containerName="revoke-keys"
Sep 12 01:09:55.215086 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.214998    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="968de32b-1dea-422a-a1aa-dc4f78412fbb" containerName="payload-pre-processing"
Sep 12 01:09:55.215086 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.215009    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdca3a0b-aea6-42f7-8550-4a0f4d23209b" containerName="payload-processing"
Sep 12 01:09:55.215086 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.215018    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9aeea73-045e-4e21-b0ce-644cf60649ad" containerName="payload-processing"
Sep 12 01:09:55.219946 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.219923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2"
Sep 12 01:09:55.223685 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.223663    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2rn9k\""
Sep 12 01:09:55.223969 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.223947    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:09:55.225912 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.225885    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2"]
Sep 12 01:09:55.333954 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.333922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f222790-5b43-4c49-b428-400ebf670c1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2\" (UID: \"5f222790-5b43-4c49-b428-400ebf670c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2"
Sep 12 01:09:55.434914 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.434877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f222790-5b43-4c49-b428-400ebf670c1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2\" (UID: \"5f222790-5b43-4c49-b428-400ebf670c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2"
Sep 12 01:09:55.435515 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.435489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f222790-5b43-4c49-b428-400ebf670c1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2\" (UID: \"5f222790-5b43-4c49-b428-400ebf670c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2"
Sep 12 01:09:55.535341 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.535267    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2rn9k\""
Sep 12 01:09:55.543537 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.543514    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2"
Sep 12 01:09:55.819265 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:55.819241    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2"]
Sep 12 01:09:55.821834 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:09:55.821809    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f222790_5b43_4c49_b428_400ebf670c1f.slice/crio-84ce6b708891240a920e481276fa09447b15b3a9c0f604c26d660264b4b2c6a5 WatchSource:0}: Error finding container 84ce6b708891240a920e481276fa09447b15b3a9c0f604c26d660264b4b2c6a5: Status 404 returned error can't find the container with id 84ce6b708891240a920e481276fa09447b15b3a9c0f604c26d660264b4b2c6a5
Sep 12 01:09:56.337168 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:56.337136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2" event={"ID":"5f222790-5b43-4c49-b428-400ebf670c1f","Type":"ContainerStarted","Data":"9decff80074f5bd6cff67ad49cc09edac6419e66e8be6680906133937aebe675"}
Sep 12 01:09:56.337551 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:56.337188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2" event={"ID":"5f222790-5b43-4c49-b428-400ebf670c1f","Type":"ContainerStarted","Data":"84ce6b708891240a920e481276fa09447b15b3a9c0f604c26d660264b4b2c6a5"}
Sep 12 01:09:56.353336 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:56.353284    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2" podStartSLOduration=1.353269534 podStartE2EDuration="1.353269534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:09:56.352179855 +0000 UTC m=+1621.271471668" watchObservedRunningTime="2026-09-12 01:09:56.353269534 +0000 UTC m=+1621.272561331"
Sep 12 01:09:57.343876 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:57.343841    2579 generic.go:356] "Generic (PLEG): container finished" podID="5f222790-5b43-4c49-b428-400ebf670c1f" containerID="9decff80074f5bd6cff67ad49cc09edac6419e66e8be6680906133937aebe675" exitCode=0
Sep 12 01:09:57.344271 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:57.343925    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2" event={"ID":"5f222790-5b43-4c49-b428-400ebf670c1f","Type":"ContainerDied","Data":"9decff80074f5bd6cff67ad49cc09edac6419e66e8be6680906133937aebe675"}
Sep 12 01:09:58.483173 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:58.483141    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2"
Sep 12 01:09:58.666458 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:58.666426    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f222790-5b43-4c49-b428-400ebf670c1f-maas-api-service-ca\") pod \"5f222790-5b43-4c49-b428-400ebf670c1f\" (UID: \"5f222790-5b43-4c49-b428-400ebf670c1f\") "
Sep 12 01:09:58.666980 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:58.666952    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f222790-5b43-4c49-b428-400ebf670c1f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f222790-5b43-4c49-b428-400ebf670c1f" (UID: "5f222790-5b43-4c49-b428-400ebf670c1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:09:58.767372 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:58.767343    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f222790-5b43-4c49-b428-400ebf670c1f-maas-api-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:09:59.355269 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:59.355237    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2"
Sep 12 01:09:59.355269 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:59.355265    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2" event={"ID":"5f222790-5b43-4c49-b428-400ebf670c1f","Type":"ContainerDied","Data":"84ce6b708891240a920e481276fa09447b15b3a9c0f604c26d660264b4b2c6a5"}
Sep 12 01:09:59.355522 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:09:59.355309    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84ce6b708891240a920e481276fa09447b15b3a9c0f604c26d660264b4b2c6a5"
Sep 12 01:10:05.779775 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:05.779741    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"]
Sep 12 01:10:05.780335 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:05.779941    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj" podUID="e26c7769-35d9-4d19-9f99-273f4193b16c" containerName="payload-processing" containerID="cri-o://23b3caeba649a1cafdd8718ca1f77dc84eb0cab72bc1b20f1b35b6d871382612" gracePeriod=30
Sep 12 01:10:05.787886 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:05.787438    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"]
Sep 12 01:10:05.787886 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:05.787618    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5" podUID="606bd754-eb44-4cb7-85c5-b68ba4115dd1" containerName="payload-pre-processing" containerID="cri-o://182e590a6f9a6f0ba0814877f17cb7483ed0acaf363591f48d41fd1608184a19" gracePeriod=30
Sep 12 01:10:06.063872 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.063845    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"
Sep 12 01:10:06.090586 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.090560    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"
Sep 12 01:10:06.128711 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.128676    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/606bd754-eb44-4cb7-85c5-b68ba4115dd1-plugins-config-volume\") pod \"606bd754-eb44-4cb7-85c5-b68ba4115dd1\" (UID: \"606bd754-eb44-4cb7-85c5-b68ba4115dd1\") "
Sep 12 01:10:06.129002 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.128976    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26c7769-35d9-4d19-9f99-273f4193b16c-plugins-config-volume\") pod \"e26c7769-35d9-4d19-9f99-273f4193b16c\" (UID: \"e26c7769-35d9-4d19-9f99-273f4193b16c\") "
Sep 12 01:10:06.129137 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.129122    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-499br\" (UniqueName: \"kubernetes.io/projected/606bd754-eb44-4cb7-85c5-b68ba4115dd1-kube-api-access-499br\") pod \"606bd754-eb44-4cb7-85c5-b68ba4115dd1\" (UID: \"606bd754-eb44-4cb7-85c5-b68ba4115dd1\") "
Sep 12 01:10:06.129303 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.129290    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtq2f\" (UniqueName: \"kubernetes.io/projected/e26c7769-35d9-4d19-9f99-273f4193b16c-kube-api-access-vtq2f\") pod \"e26c7769-35d9-4d19-9f99-273f4193b16c\" (UID: \"e26c7769-35d9-4d19-9f99-273f4193b16c\") "
Sep 12 01:10:06.130411 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.130383    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26c7769-35d9-4d19-9f99-273f4193b16c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e26c7769-35d9-4d19-9f99-273f4193b16c" (UID: "e26c7769-35d9-4d19-9f99-273f4193b16c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:10:06.131012 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.130989    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606bd754-eb44-4cb7-85c5-b68ba4115dd1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "606bd754-eb44-4cb7-85c5-b68ba4115dd1" (UID: "606bd754-eb44-4cb7-85c5-b68ba4115dd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:10:06.133854 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.133831    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606bd754-eb44-4cb7-85c5-b68ba4115dd1-kube-api-access-499br" (OuterVolumeSpecName: "kube-api-access-499br") pod "606bd754-eb44-4cb7-85c5-b68ba4115dd1" (UID: "606bd754-eb44-4cb7-85c5-b68ba4115dd1"). InnerVolumeSpecName "kube-api-access-499br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:10:06.135635 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.135582    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26c7769-35d9-4d19-9f99-273f4193b16c-kube-api-access-vtq2f" (OuterVolumeSpecName: "kube-api-access-vtq2f") pod "e26c7769-35d9-4d19-9f99-273f4193b16c" (UID: "e26c7769-35d9-4d19-9f99-273f4193b16c"). InnerVolumeSpecName "kube-api-access-vtq2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:10:06.230856 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.230816    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/606bd754-eb44-4cb7-85c5-b68ba4115dd1-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:06.230856 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.230857    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26c7769-35d9-4d19-9f99-273f4193b16c-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:06.231054 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.230872    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-499br\" (UniqueName: \"kubernetes.io/projected/606bd754-eb44-4cb7-85c5-b68ba4115dd1-kube-api-access-499br\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:06.231054 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.230886    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtq2f\" (UniqueName: \"kubernetes.io/projected/e26c7769-35d9-4d19-9f99-273f4193b16c-kube-api-access-vtq2f\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:06.397972 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.397881    2579 generic.go:356] "Generic (PLEG): container finished" podID="606bd754-eb44-4cb7-85c5-b68ba4115dd1" containerID="182e590a6f9a6f0ba0814877f17cb7483ed0acaf363591f48d41fd1608184a19" exitCode=0
Sep 12 01:10:06.397972 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.397951    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"
Sep 12 01:10:06.398203 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.397964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5" event={"ID":"606bd754-eb44-4cb7-85c5-b68ba4115dd1","Type":"ContainerDied","Data":"182e590a6f9a6f0ba0814877f17cb7483ed0acaf363591f48d41fd1608184a19"}
Sep 12 01:10:06.398203 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.398059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5" event={"ID":"606bd754-eb44-4cb7-85c5-b68ba4115dd1","Type":"ContainerDied","Data":"136fb6b7d522b2db68e9fa5bbb7b6765626b45e027fdd8b5fb89a5432748d294"}
Sep 12 01:10:06.398203 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.398094    2579 scope.go:117] "RemoveContainer" containerID="182e590a6f9a6f0ba0814877f17cb7483ed0acaf363591f48d41fd1608184a19"
Sep 12 01:10:06.400128 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.400108    2579 generic.go:356] "Generic (PLEG): container finished" podID="e26c7769-35d9-4d19-9f99-273f4193b16c" containerID="23b3caeba649a1cafdd8718ca1f77dc84eb0cab72bc1b20f1b35b6d871382612" exitCode=0
Sep 12 01:10:06.400303 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.400212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj" event={"ID":"e26c7769-35d9-4d19-9f99-273f4193b16c","Type":"ContainerDied","Data":"23b3caeba649a1cafdd8718ca1f77dc84eb0cab72bc1b20f1b35b6d871382612"}
Sep 12 01:10:06.400403 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.400333    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj" event={"ID":"e26c7769-35d9-4d19-9f99-273f4193b16c","Type":"ContainerDied","Data":"b09d2be8dde8e8731f83ba3178daf3c6a63741af6e8f5cae7e6fe44739328bd0"}
Sep 12 01:10:06.400403 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.400232    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"
Sep 12 01:10:06.410180 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.410087    2579 scope.go:117] "RemoveContainer" containerID="182e590a6f9a6f0ba0814877f17cb7483ed0acaf363591f48d41fd1608184a19"
Sep 12 01:10:06.410472 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:10:06.410446    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"182e590a6f9a6f0ba0814877f17cb7483ed0acaf363591f48d41fd1608184a19\": container with ID starting with 182e590a6f9a6f0ba0814877f17cb7483ed0acaf363591f48d41fd1608184a19 not found: ID does not exist" containerID="182e590a6f9a6f0ba0814877f17cb7483ed0acaf363591f48d41fd1608184a19"
Sep 12 01:10:06.410573 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.410483    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"182e590a6f9a6f0ba0814877f17cb7483ed0acaf363591f48d41fd1608184a19"} err="failed to get container status \"182e590a6f9a6f0ba0814877f17cb7483ed0acaf363591f48d41fd1608184a19\": rpc error: code = NotFound desc = could not find container \"182e590a6f9a6f0ba0814877f17cb7483ed0acaf363591f48d41fd1608184a19\": container with ID starting with 182e590a6f9a6f0ba0814877f17cb7483ed0acaf363591f48d41fd1608184a19 not found: ID does not exist"
Sep 12 01:10:06.410573 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.410505    2579 scope.go:117] "RemoveContainer" containerID="23b3caeba649a1cafdd8718ca1f77dc84eb0cab72bc1b20f1b35b6d871382612"
Sep 12 01:10:06.418960 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.418941    2579 scope.go:117] "RemoveContainer" containerID="23b3caeba649a1cafdd8718ca1f77dc84eb0cab72bc1b20f1b35b6d871382612"
Sep 12 01:10:06.419218 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:10:06.419199    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b3caeba649a1cafdd8718ca1f77dc84eb0cab72bc1b20f1b35b6d871382612\": container with ID starting with 23b3caeba649a1cafdd8718ca1f77dc84eb0cab72bc1b20f1b35b6d871382612 not found: ID does not exist" containerID="23b3caeba649a1cafdd8718ca1f77dc84eb0cab72bc1b20f1b35b6d871382612"
Sep 12 01:10:06.419285 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.419228    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b3caeba649a1cafdd8718ca1f77dc84eb0cab72bc1b20f1b35b6d871382612"} err="failed to get container status \"23b3caeba649a1cafdd8718ca1f77dc84eb0cab72bc1b20f1b35b6d871382612\": rpc error: code = NotFound desc = could not find container \"23b3caeba649a1cafdd8718ca1f77dc84eb0cab72bc1b20f1b35b6d871382612\": container with ID starting with 23b3caeba649a1cafdd8718ca1f77dc84eb0cab72bc1b20f1b35b6d871382612 not found: ID does not exist"
Sep 12 01:10:06.427294 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.427274    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"]
Sep 12 01:10:06.433759 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.433737    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-18b9d7-6bf785d4df-t57tj"]
Sep 12 01:10:06.444342 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.444322    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"]
Sep 12 01:10:06.446984 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:06.446965    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-18b9d7-68887bbd94-27km5"]
Sep 12 01:10:07.591849 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:07.591818    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="606bd754-eb44-4cb7-85c5-b68ba4115dd1" path="/var/lib/kubelet/pods/606bd754-eb44-4cb7-85c5-b68ba4115dd1/volumes"
Sep 12 01:10:07.592209 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:07.592165    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e26c7769-35d9-4d19-9f99-273f4193b16c" path="/var/lib/kubelet/pods/e26c7769-35d9-4d19-9f99-273f4193b16c/volumes"
Sep 12 01:10:08.111766 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.111733    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"]
Sep 12 01:10:08.111981 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.111959    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb" podUID="4d0c2aa1-7e24-4d5b-a711-1f8657dbe378" containerName="payload-processing" containerID="cri-o://cdd46b6559a52acb821c4202c9146e6b781c9583d00feff73f8a99641906d544" gracePeriod=30
Sep 12 01:10:08.116745 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.116704    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"]
Sep 12 01:10:08.116969 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.116929    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p" podUID="d98b6109-80a7-4ad8-8795-47430bea0a98" containerName="payload-pre-processing" containerID="cri-o://f6e860014bf65138a45b53978bab43efbcb98a105792281c42ea114f5b67f94a" gracePeriod=30
Sep 12 01:10:08.356166 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.356096    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2"]
Sep 12 01:10:08.358921 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.358893    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f19fd1-6hqm2"]
Sep 12 01:10:08.381139 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.381120    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"
Sep 12 01:10:08.415024 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.414996    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"
Sep 12 01:10:08.416250 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.416224    2579 generic.go:356] "Generic (PLEG): container finished" podID="4d0c2aa1-7e24-4d5b-a711-1f8657dbe378" containerID="cdd46b6559a52acb821c4202c9146e6b781c9583d00feff73f8a99641906d544" exitCode=0
Sep 12 01:10:08.416384 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.416262    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"
Sep 12 01:10:08.416384 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.416312    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb" event={"ID":"4d0c2aa1-7e24-4d5b-a711-1f8657dbe378","Type":"ContainerDied","Data":"cdd46b6559a52acb821c4202c9146e6b781c9583d00feff73f8a99641906d544"}
Sep 12 01:10:08.416384 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.416369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb" event={"ID":"4d0c2aa1-7e24-4d5b-a711-1f8657dbe378","Type":"ContainerDied","Data":"8259ca8015f9a06205a4f8451e9091ad008ffa2fa0aabc7ed2b0e06d0c8273e6"}
Sep 12 01:10:08.416557 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.416390    2579 scope.go:117] "RemoveContainer" containerID="cdd46b6559a52acb821c4202c9146e6b781c9583d00feff73f8a99641906d544"
Sep 12 01:10:08.418094 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.418074    2579 generic.go:356] "Generic (PLEG): container finished" podID="d98b6109-80a7-4ad8-8795-47430bea0a98" containerID="f6e860014bf65138a45b53978bab43efbcb98a105792281c42ea114f5b67f94a" exitCode=0
Sep 12 01:10:08.418208 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.418156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p" event={"ID":"d98b6109-80a7-4ad8-8795-47430bea0a98","Type":"ContainerDied","Data":"f6e860014bf65138a45b53978bab43efbcb98a105792281c42ea114f5b67f94a"}
Sep 12 01:10:08.418270 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.418204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p" event={"ID":"d98b6109-80a7-4ad8-8795-47430bea0a98","Type":"ContainerDied","Data":"97b67ac3e60098e93daeef253e0296763b86b131357e1399c76e6b4a6c768677"}
Sep 12 01:10:08.418326 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.418273    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"
Sep 12 01:10:08.427671 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.427651    2579 scope.go:117] "RemoveContainer" containerID="cdd46b6559a52acb821c4202c9146e6b781c9583d00feff73f8a99641906d544"
Sep 12 01:10:08.427952 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:10:08.427930    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdd46b6559a52acb821c4202c9146e6b781c9583d00feff73f8a99641906d544\": container with ID starting with cdd46b6559a52acb821c4202c9146e6b781c9583d00feff73f8a99641906d544 not found: ID does not exist" containerID="cdd46b6559a52acb821c4202c9146e6b781c9583d00feff73f8a99641906d544"
Sep 12 01:10:08.428011 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.427959    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdd46b6559a52acb821c4202c9146e6b781c9583d00feff73f8a99641906d544"} err="failed to get container status \"cdd46b6559a52acb821c4202c9146e6b781c9583d00feff73f8a99641906d544\": rpc error: code = NotFound desc = could not find container \"cdd46b6559a52acb821c4202c9146e6b781c9583d00feff73f8a99641906d544\": container with ID starting with cdd46b6559a52acb821c4202c9146e6b781c9583d00feff73f8a99641906d544 not found: ID does not exist"
Sep 12 01:10:08.428011 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.427978    2579 scope.go:117] "RemoveContainer" containerID="f6e860014bf65138a45b53978bab43efbcb98a105792281c42ea114f5b67f94a"
Sep 12 01:10:08.437518 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.437503    2579 scope.go:117] "RemoveContainer" containerID="f6e860014bf65138a45b53978bab43efbcb98a105792281c42ea114f5b67f94a"
Sep 12 01:10:08.437883 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:10:08.437863    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e860014bf65138a45b53978bab43efbcb98a105792281c42ea114f5b67f94a\": container with ID starting with f6e860014bf65138a45b53978bab43efbcb98a105792281c42ea114f5b67f94a not found: ID does not exist" containerID="f6e860014bf65138a45b53978bab43efbcb98a105792281c42ea114f5b67f94a"
Sep 12 01:10:08.437964 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.437890    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e860014bf65138a45b53978bab43efbcb98a105792281c42ea114f5b67f94a"} err="failed to get container status \"f6e860014bf65138a45b53978bab43efbcb98a105792281c42ea114f5b67f94a\": rpc error: code = NotFound desc = could not find container \"f6e860014bf65138a45b53978bab43efbcb98a105792281c42ea114f5b67f94a\": container with ID starting with f6e860014bf65138a45b53978bab43efbcb98a105792281c42ea114f5b67f94a not found: ID does not exist"
Sep 12 01:10:08.452773 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.452733    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98b6109-80a7-4ad8-8795-47430bea0a98-plugins-config-volume\") pod \"d98b6109-80a7-4ad8-8795-47430bea0a98\" (UID: \"d98b6109-80a7-4ad8-8795-47430bea0a98\") "
Sep 12 01:10:08.453438 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.453418    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n2cj\" (UniqueName: \"kubernetes.io/projected/d98b6109-80a7-4ad8-8795-47430bea0a98-kube-api-access-8n2cj\") pod \"d98b6109-80a7-4ad8-8795-47430bea0a98\" (UID: \"d98b6109-80a7-4ad8-8795-47430bea0a98\") "
Sep 12 01:10:08.453924 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.453906    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0c2aa1-7e24-4d5b-a711-1f8657dbe378-plugins-config-volume\") pod \"4d0c2aa1-7e24-4d5b-a711-1f8657dbe378\" (UID: \"4d0c2aa1-7e24-4d5b-a711-1f8657dbe378\") "
Sep 12 01:10:08.454304 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.453305    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98b6109-80a7-4ad8-8795-47430bea0a98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d98b6109-80a7-4ad8-8795-47430bea0a98" (UID: "d98b6109-80a7-4ad8-8795-47430bea0a98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:10:08.454391 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.454311    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0c2aa1-7e24-4d5b-a711-1f8657dbe378-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d0c2aa1-7e24-4d5b-a711-1f8657dbe378" (UID: "4d0c2aa1-7e24-4d5b-a711-1f8657dbe378"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:10:08.454546 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.454515    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrjx2\" (UniqueName: \"kubernetes.io/projected/4d0c2aa1-7e24-4d5b-a711-1f8657dbe378-kube-api-access-vrjx2\") pod \"4d0c2aa1-7e24-4d5b-a711-1f8657dbe378\" (UID: \"4d0c2aa1-7e24-4d5b-a711-1f8657dbe378\") "
Sep 12 01:10:08.454825 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.454804    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d0c2aa1-7e24-4d5b-a711-1f8657dbe378-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:08.454897 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.454831    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98b6109-80a7-4ad8-8795-47430bea0a98-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:08.455921 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.455890    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98b6109-80a7-4ad8-8795-47430bea0a98-kube-api-access-8n2cj" (OuterVolumeSpecName: "kube-api-access-8n2cj") pod "d98b6109-80a7-4ad8-8795-47430bea0a98" (UID: "d98b6109-80a7-4ad8-8795-47430bea0a98"). InnerVolumeSpecName "kube-api-access-8n2cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:10:08.456768 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.456749    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0c2aa1-7e24-4d5b-a711-1f8657dbe378-kube-api-access-vrjx2" (OuterVolumeSpecName: "kube-api-access-vrjx2") pod "4d0c2aa1-7e24-4d5b-a711-1f8657dbe378" (UID: "4d0c2aa1-7e24-4d5b-a711-1f8657dbe378"). InnerVolumeSpecName "kube-api-access-vrjx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:10:08.556043 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.556013    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n2cj\" (UniqueName: \"kubernetes.io/projected/d98b6109-80a7-4ad8-8795-47430bea0a98-kube-api-access-8n2cj\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:08.556043 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.556041    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrjx2\" (UniqueName: \"kubernetes.io/projected/4d0c2aa1-7e24-4d5b-a711-1f8657dbe378-kube-api-access-vrjx2\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:08.742709 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.742679    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"]
Sep 12 01:10:08.746054 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.746032    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-f19fd1-5c9c988f49-5fjvb"]
Sep 12 01:10:08.758193 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.758171    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"]
Sep 12 01:10:08.759901 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:08.759879    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-f19fd1-6647b85cfd-9vw6p"]
Sep 12 01:10:09.591235 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:09.591204    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d0c2aa1-7e24-4d5b-a711-1f8657dbe378" path="/var/lib/kubelet/pods/4d0c2aa1-7e24-4d5b-a711-1f8657dbe378/volumes"
Sep 12 01:10:09.591578 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:09.591563    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f222790-5b43-4c49-b428-400ebf670c1f" path="/var/lib/kubelet/pods/5f222790-5b43-4c49-b428-400ebf670c1f/volumes"
Sep 12 01:10:09.591915 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:09.591902    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d98b6109-80a7-4ad8-8795-47430bea0a98" path="/var/lib/kubelet/pods/d98b6109-80a7-4ad8-8795-47430bea0a98/volumes"
Sep 12 01:10:13.014374 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:13.014341    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"]
Sep 12 01:10:13.014788 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:13.014616    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m" podUID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" containerName="istio-proxy" containerID="cri-o://31eaf5a275a3d0a23eae5925231f7aba5461a86331a95458283b761e5e434ed3" gracePeriod=30
Sep 12 01:10:15.190412 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:15.190368    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"]
Sep 12 01:10:15.190892 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:15.190634    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc" podUID="171640c8-513b-4b58-a865-08d1cf2faf75" containerName="istio-proxy" containerID="cri-o://5c837d175f9d150aab35766308abda123cb0e9e4e228672a754c4f0bb7a408e2" gracePeriod=30
Sep 12 01:10:18.267806 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.267782    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:10:18.341668 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.341569    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-workload-socket\") pod \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") "
Sep 12 01:10:18.341668 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.341638    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-workload-certs\") pod \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") "
Sep 12 01:10:18.341874 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.341668    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-token\") pod \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") "
Sep 12 01:10:18.341874 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.341705    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istiod-ca-cert\") pod \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") "
Sep 12 01:10:18.341874 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.341738    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rkqq\" (UniqueName: \"kubernetes.io/projected/fa0b453e-3223-4a85-956d-d868ea6e4e9f-kube-api-access-5rkqq\") pod \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") "
Sep 12 01:10:18.341874 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.341793    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-data\") pod \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") "
Sep 12 01:10:18.341874 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.341827    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-podinfo\") pod \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") "
Sep 12 01:10:18.341874 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.341853    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fa0b453e-3223-4a85-956d-d868ea6e4e9f" (UID: "fa0b453e-3223-4a85-956d-d868ea6e4e9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:18.342186 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.341885    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-credential-socket\") pod \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") "
Sep 12 01:10:18.342186 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.341919    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-envoy\") pod \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\" (UID: \"fa0b453e-3223-4a85-956d-d868ea6e4e9f\") "
Sep 12 01:10:18.342186 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.341948    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fa0b453e-3223-4a85-956d-d868ea6e4e9f" (UID: "fa0b453e-3223-4a85-956d-d868ea6e4e9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:18.342186 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.342068    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fa0b453e-3223-4a85-956d-d868ea6e4e9f" (UID: "fa0b453e-3223-4a85-956d-d868ea6e4e9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:10:18.342186 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.342081    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-data" (OuterVolumeSpecName: "istio-data") pod "fa0b453e-3223-4a85-956d-d868ea6e4e9f" (UID: "fa0b453e-3223-4a85-956d-d868ea6e4e9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:18.342422 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.342218    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:18.342422 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.342232    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:18.342422 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.342240    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:18.342422 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.342249    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:18.342422 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.342253    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fa0b453e-3223-4a85-956d-d868ea6e4e9f" (UID: "fa0b453e-3223-4a85-956d-d868ea6e4e9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:18.343983 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.343958    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fa0b453e-3223-4a85-956d-d868ea6e4e9f" (UID: "fa0b453e-3223-4a85-956d-d868ea6e4e9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 01:10:18.344121 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.343999    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fa0b453e-3223-4a85-956d-d868ea6e4e9f" (UID: "fa0b453e-3223-4a85-956d-d868ea6e4e9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:18.344278 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.344256    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0b453e-3223-4a85-956d-d868ea6e4e9f-kube-api-access-5rkqq" (OuterVolumeSpecName: "kube-api-access-5rkqq") pod "fa0b453e-3223-4a85-956d-d868ea6e4e9f" (UID: "fa0b453e-3223-4a85-956d-d868ea6e4e9f"). InnerVolumeSpecName "kube-api-access-5rkqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:10:18.344365 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.344308    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-token" (OuterVolumeSpecName: "istio-token") pod "fa0b453e-3223-4a85-956d-d868ea6e4e9f" (UID: "fa0b453e-3223-4a85-956d-d868ea6e4e9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:10:18.442593 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.442565    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rkqq\" (UniqueName: \"kubernetes.io/projected/fa0b453e-3223-4a85-956d-d868ea6e4e9f-kube-api-access-5rkqq\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:18.442593 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.442587    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:18.442593 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.442596    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:18.442823 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.442617    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:18.442823 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.442625    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa0b453e-3223-4a85-956d-d868ea6e4e9f-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:18.471300 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.471269    2579 generic.go:356] "Generic (PLEG): container finished" podID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" containerID="31eaf5a275a3d0a23eae5925231f7aba5461a86331a95458283b761e5e434ed3" exitCode=0
Sep 12 01:10:18.471412 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.471315    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"
Sep 12 01:10:18.471412 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.471343    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m" event={"ID":"fa0b453e-3223-4a85-956d-d868ea6e4e9f","Type":"ContainerDied","Data":"31eaf5a275a3d0a23eae5925231f7aba5461a86331a95458283b761e5e434ed3"}
Sep 12 01:10:18.471412 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.471383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m" event={"ID":"fa0b453e-3223-4a85-956d-d868ea6e4e9f","Type":"ContainerDied","Data":"ebb3c9b5ae7a9545c2e715bc782dc268e29f7540ebb80e681983c872af025bef"}
Sep 12 01:10:18.471412 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.471404    2579 scope.go:117] "RemoveContainer" containerID="31eaf5a275a3d0a23eae5925231f7aba5461a86331a95458283b761e5e434ed3"
Sep 12 01:10:18.480036 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.480021    2579 scope.go:117] "RemoveContainer" containerID="faed5a5ac487a74d95c01eea8839017c32964c07c6ca361174bbd6d8961ff788"
Sep 12 01:10:18.487418 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.487396    2579 scope.go:117] "RemoveContainer" containerID="31eaf5a275a3d0a23eae5925231f7aba5461a86331a95458283b761e5e434ed3"
Sep 12 01:10:18.487691 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:10:18.487672    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31eaf5a275a3d0a23eae5925231f7aba5461a86331a95458283b761e5e434ed3\": container with ID starting with 31eaf5a275a3d0a23eae5925231f7aba5461a86331a95458283b761e5e434ed3 not found: ID does not exist" containerID="31eaf5a275a3d0a23eae5925231f7aba5461a86331a95458283b761e5e434ed3"
Sep 12 01:10:18.487761 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.487700    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31eaf5a275a3d0a23eae5925231f7aba5461a86331a95458283b761e5e434ed3"} err="failed to get container status \"31eaf5a275a3d0a23eae5925231f7aba5461a86331a95458283b761e5e434ed3\": rpc error: code = NotFound desc = could not find container \"31eaf5a275a3d0a23eae5925231f7aba5461a86331a95458283b761e5e434ed3\": container with ID starting with 31eaf5a275a3d0a23eae5925231f7aba5461a86331a95458283b761e5e434ed3 not found: ID does not exist"
Sep 12 01:10:18.487761 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.487720    2579 scope.go:117] "RemoveContainer" containerID="faed5a5ac487a74d95c01eea8839017c32964c07c6ca361174bbd6d8961ff788"
Sep 12 01:10:18.487984 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:10:18.487971    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faed5a5ac487a74d95c01eea8839017c32964c07c6ca361174bbd6d8961ff788\": container with ID starting with faed5a5ac487a74d95c01eea8839017c32964c07c6ca361174bbd6d8961ff788 not found: ID does not exist" containerID="faed5a5ac487a74d95c01eea8839017c32964c07c6ca361174bbd6d8961ff788"
Sep 12 01:10:18.488035 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.487986    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faed5a5ac487a74d95c01eea8839017c32964c07c6ca361174bbd6d8961ff788"} err="failed to get container status \"faed5a5ac487a74d95c01eea8839017c32964c07c6ca361174bbd6d8961ff788\": rpc error: code = NotFound desc = could not find container \"faed5a5ac487a74d95c01eea8839017c32964c07c6ca361174bbd6d8961ff788\": container with ID starting with faed5a5ac487a74d95c01eea8839017c32964c07c6ca361174bbd6d8961ff788 not found: ID does not exist"
Sep 12 01:10:18.494709 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.494689    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"]
Sep 12 01:10:18.500891 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:18.500872    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-18b9d7-openshift-default-68d959d74-2bl5m"]
Sep 12 01:10:19.591179 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:19.591151    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" path="/var/lib/kubelet/pods/fa0b453e-3223-4a85-956d-d868ea6e4e9f/volumes"
Sep 12 01:10:20.446539 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.446515    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 01:10:20.484702 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.484624    2579 generic.go:356] "Generic (PLEG): container finished" podID="171640c8-513b-4b58-a865-08d1cf2faf75" containerID="5c837d175f9d150aab35766308abda123cb0e9e4e228672a754c4f0bb7a408e2" exitCode=0
Sep 12 01:10:20.484702 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.484670    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"
Sep 12 01:10:20.484884 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.484696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc" event={"ID":"171640c8-513b-4b58-a865-08d1cf2faf75","Type":"ContainerDied","Data":"5c837d175f9d150aab35766308abda123cb0e9e4e228672a754c4f0bb7a408e2"}
Sep 12 01:10:20.484884 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.484749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc" event={"ID":"171640c8-513b-4b58-a865-08d1cf2faf75","Type":"ContainerDied","Data":"a083122083854642b6e2b3c74a8a71c02ae06d4edb2e8067c94dd67a5127cf06"}
Sep 12 01:10:20.484884 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.484772    2579 scope.go:117] "RemoveContainer" containerID="5c837d175f9d150aab35766308abda123cb0e9e4e228672a754c4f0bb7a408e2"
Sep 12 01:10:20.493093 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.493068    2579 scope.go:117] "RemoveContainer" containerID="771af17de8c7920efbfb1df998f77e8819bc1a4618296cd30333f3b9d971f685"
Sep 12 01:10:20.500740 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.500721    2579 scope.go:117] "RemoveContainer" containerID="5c837d175f9d150aab35766308abda123cb0e9e4e228672a754c4f0bb7a408e2"
Sep 12 01:10:20.501071 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:10:20.501043    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c837d175f9d150aab35766308abda123cb0e9e4e228672a754c4f0bb7a408e2\": container with ID starting with 5c837d175f9d150aab35766308abda123cb0e9e4e228672a754c4f0bb7a408e2 not found: ID does not exist" containerID="5c837d175f9d150aab35766308abda123cb0e9e4e228672a754c4f0bb7a408e2"
Sep 12 01:10:20.501162 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.501073    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c837d175f9d150aab35766308abda123cb0e9e4e228672a754c4f0bb7a408e2"} err="failed to get container status \"5c837d175f9d150aab35766308abda123cb0e9e4e228672a754c4f0bb7a408e2\": rpc error: code = NotFound desc = could not find container \"5c837d175f9d150aab35766308abda123cb0e9e4e228672a754c4f0bb7a408e2\": container with ID starting with 5c837d175f9d150aab35766308abda123cb0e9e4e228672a754c4f0bb7a408e2 not found: ID does not exist"
Sep 12 01:10:20.501162 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.501098    2579 scope.go:117] "RemoveContainer" containerID="771af17de8c7920efbfb1df998f77e8819bc1a4618296cd30333f3b9d971f685"
Sep 12 01:10:20.501397 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:10:20.501379    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771af17de8c7920efbfb1df998f77e8819bc1a4618296cd30333f3b9d971f685\": container with ID starting with 771af17de8c7920efbfb1df998f77e8819bc1a4618296cd30333f3b9d971f685 not found: ID does not exist" containerID="771af17de8c7920efbfb1df998f77e8819bc1a4618296cd30333f3b9d971f685"
Sep 12 01:10:20.501454 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.501405    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771af17de8c7920efbfb1df998f77e8819bc1a4618296cd30333f3b9d971f685"} err="failed to get container status \"771af17de8c7920efbfb1df998f77e8819bc1a4618296cd30333f3b9d971f685\": rpc error: code = NotFound desc = could not find container \"771af17de8c7920efbfb1df998f77e8819bc1a4618296cd30333f3b9d971f685\": container with ID starting with 771af17de8c7920efbfb1df998f77e8819bc1a4618296cd30333f3b9d971f685 not found: ID does not exist"
Sep 12 01:10:20.563607 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.563583    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-istio-envoy\") pod \"171640c8-513b-4b58-a865-08d1cf2faf75\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") "
Sep 12 01:10:20.563721 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.563635    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-istio-data\") pod \"171640c8-513b-4b58-a865-08d1cf2faf75\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") "
Sep 12 01:10:20.563721 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.563654    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-workload-certs\") pod \"171640c8-513b-4b58-a865-08d1cf2faf75\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") "
Sep 12 01:10:20.563721 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.563672    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-credential-socket\") pod \"171640c8-513b-4b58-a865-08d1cf2faf75\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") "
Sep 12 01:10:20.563721 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.563696    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8v59\" (UniqueName: \"kubernetes.io/projected/171640c8-513b-4b58-a865-08d1cf2faf75-kube-api-access-g8v59\") pod \"171640c8-513b-4b58-a865-08d1cf2faf75\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") "
Sep 12 01:10:20.563931 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.563743    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-workload-socket\") pod \"171640c8-513b-4b58-a865-08d1cf2faf75\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") "
Sep 12 01:10:20.563931 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.563762    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171640c8-513b-4b58-a865-08d1cf2faf75-istio-token\") pod \"171640c8-513b-4b58-a865-08d1cf2faf75\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") "
Sep 12 01:10:20.563931 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.563817    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171640c8-513b-4b58-a865-08d1cf2faf75-istiod-ca-cert\") pod \"171640c8-513b-4b58-a865-08d1cf2faf75\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") "
Sep 12 01:10:20.563931 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.563847    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171640c8-513b-4b58-a865-08d1cf2faf75-istio-podinfo\") pod \"171640c8-513b-4b58-a865-08d1cf2faf75\" (UID: \"171640c8-513b-4b58-a865-08d1cf2faf75\") "
Sep 12 01:10:20.563931 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.563908    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-istio-data" (OuterVolumeSpecName: "istio-data") pod "171640c8-513b-4b58-a865-08d1cf2faf75" (UID: "171640c8-513b-4b58-a865-08d1cf2faf75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:20.563931 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.563920    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "171640c8-513b-4b58-a865-08d1cf2faf75" (UID: "171640c8-513b-4b58-a865-08d1cf2faf75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:20.564240 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.563944    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "171640c8-513b-4b58-a865-08d1cf2faf75" (UID: "171640c8-513b-4b58-a865-08d1cf2faf75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:20.564240 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.564058    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "171640c8-513b-4b58-a865-08d1cf2faf75" (UID: "171640c8-513b-4b58-a865-08d1cf2faf75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:20.564240 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.564172    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:20.564240 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.564211    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:20.564240 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.564226    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:20.564240 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.564227    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171640c8-513b-4b58-a865-08d1cf2faf75-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "171640c8-513b-4b58-a865-08d1cf2faf75" (UID: "171640c8-513b-4b58-a865-08d1cf2faf75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:10:20.564240 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.564240    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:20.565721 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.565703    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "171640c8-513b-4b58-a865-08d1cf2faf75" (UID: "171640c8-513b-4b58-a865-08d1cf2faf75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:10:20.565820 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.565806    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171640c8-513b-4b58-a865-08d1cf2faf75-istio-token" (OuterVolumeSpecName: "istio-token") pod "171640c8-513b-4b58-a865-08d1cf2faf75" (UID: "171640c8-513b-4b58-a865-08d1cf2faf75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:10:20.566192 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.566175    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171640c8-513b-4b58-a865-08d1cf2faf75-kube-api-access-g8v59" (OuterVolumeSpecName: "kube-api-access-g8v59") pod "171640c8-513b-4b58-a865-08d1cf2faf75" (UID: "171640c8-513b-4b58-a865-08d1cf2faf75"). InnerVolumeSpecName "kube-api-access-g8v59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:10:20.566251 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.566187    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/171640c8-513b-4b58-a865-08d1cf2faf75-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "171640c8-513b-4b58-a865-08d1cf2faf75" (UID: "171640c8-513b-4b58-a865-08d1cf2faf75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 01:10:20.664850 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.664817    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/171640c8-513b-4b58-a865-08d1cf2faf75-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:20.664850 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.664846    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8v59\" (UniqueName: \"kubernetes.io/projected/171640c8-513b-4b58-a865-08d1cf2faf75-kube-api-access-g8v59\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:20.665217 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.664860    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/171640c8-513b-4b58-a865-08d1cf2faf75-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:20.665217 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.664871    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/171640c8-513b-4b58-a865-08d1cf2faf75-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:20.665217 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.664882    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/171640c8-513b-4b58-a865-08d1cf2faf75-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:10:20.806963 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.806930    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"]
Sep 12 01:10:20.811879 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:20.811856    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f19fd1-openshift-default-74f6d7f46b-8tpgc"]
Sep 12 01:10:21.590897 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:21.590862    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="171640c8-513b-4b58-a865-08d1cf2faf75" path="/var/lib/kubelet/pods/171640c8-513b-4b58-a865-08d1cf2faf75/volumes"
Sep 12 01:10:46.339430 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:10:46.339389    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"]
Sep 12 01:11:01.357731 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.357671    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l" podUID="bf9f0c29-9c94-47c2-bc4e-09ad14d3041e" containerName="main" containerID="cri-o://6b05cb463042a2d756c62db149db9a1150af1e92d97a5e9fb076f83c4c8c2692" gracePeriod=45
Sep 12 01:11:01.603986 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.603960    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:11:01.688944 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.688916    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-tmp-dir\") pod \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") "
Sep 12 01:11:01.689097 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.688973    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-model-cache\") pod \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") "
Sep 12 01:11:01.689097 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.688998    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-dshm\") pod \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") "
Sep 12 01:11:01.689097 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.689038    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-home\") pod \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") "
Sep 12 01:11:01.689097 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.689086    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnfjm\" (UniqueName: \"kubernetes.io/projected/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-kube-api-access-dnfjm\") pod \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") "
Sep 12 01:11:01.689284 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.689118    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-tls-certs\") pod \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\" (UID: \"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e\") "
Sep 12 01:11:01.689284 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.689192    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf9f0c29-9c94-47c2-bc4e-09ad14d3041e" (UID: "bf9f0c29-9c94-47c2-bc4e-09ad14d3041e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:11:01.689284 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.689214    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf9f0c29-9c94-47c2-bc4e-09ad14d3041e" (UID: "bf9f0c29-9c94-47c2-bc4e-09ad14d3041e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:11:01.689456 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.689292    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-home" (OuterVolumeSpecName: "home") pod "bf9f0c29-9c94-47c2-bc4e-09ad14d3041e" (UID: "bf9f0c29-9c94-47c2-bc4e-09ad14d3041e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:11:01.689456 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.689337    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-model-cache\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:11:01.689456 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.689350    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-tmp-dir\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:11:01.691161 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.691137    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-dshm" (OuterVolumeSpecName: "dshm") pod "bf9f0c29-9c94-47c2-bc4e-09ad14d3041e" (UID: "bf9f0c29-9c94-47c2-bc4e-09ad14d3041e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:11:01.691269 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.691241    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-kube-api-access-dnfjm" (OuterVolumeSpecName: "kube-api-access-dnfjm") pod "bf9f0c29-9c94-47c2-bc4e-09ad14d3041e" (UID: "bf9f0c29-9c94-47c2-bc4e-09ad14d3041e"). InnerVolumeSpecName "kube-api-access-dnfjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:11:01.691319 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.691292    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf9f0c29-9c94-47c2-bc4e-09ad14d3041e" (UID: "bf9f0c29-9c94-47c2-bc4e-09ad14d3041e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:11:01.700972 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.700948    2579 generic.go:356] "Generic (PLEG): container finished" podID="bf9f0c29-9c94-47c2-bc4e-09ad14d3041e" containerID="6b05cb463042a2d756c62db149db9a1150af1e92d97a5e9fb076f83c4c8c2692" exitCode=0
Sep 12 01:11:01.701083 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.701021    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"
Sep 12 01:11:01.701083 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.701036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l" event={"ID":"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e","Type":"ContainerDied","Data":"6b05cb463042a2d756c62db149db9a1150af1e92d97a5e9fb076f83c4c8c2692"}
Sep 12 01:11:01.701187 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.701082    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l" event={"ID":"bf9f0c29-9c94-47c2-bc4e-09ad14d3041e","Type":"ContainerDied","Data":"ef191c62b61ea710b7d63097773374e9a48576447cf4e87b3b982f4accb6dd6b"}
Sep 12 01:11:01.701187 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.701107    2579 scope.go:117] "RemoveContainer" containerID="6b05cb463042a2d756c62db149db9a1150af1e92d97a5e9fb076f83c4c8c2692"
Sep 12 01:11:01.710886 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.710867    2579 scope.go:117] "RemoveContainer" containerID="6b05cb463042a2d756c62db149db9a1150af1e92d97a5e9fb076f83c4c8c2692"
Sep 12 01:11:01.711192 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:11:01.711165    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b05cb463042a2d756c62db149db9a1150af1e92d97a5e9fb076f83c4c8c2692\": container with ID starting with 6b05cb463042a2d756c62db149db9a1150af1e92d97a5e9fb076f83c4c8c2692 not found: ID does not exist" containerID="6b05cb463042a2d756c62db149db9a1150af1e92d97a5e9fb076f83c4c8c2692"
Sep 12 01:11:01.711263 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.711199    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b05cb463042a2d756c62db149db9a1150af1e92d97a5e9fb076f83c4c8c2692"} err="failed to get container status \"6b05cb463042a2d756c62db149db9a1150af1e92d97a5e9fb076f83c4c8c2692\": rpc error: code = NotFound desc = could not find container \"6b05cb463042a2d756c62db149db9a1150af1e92d97a5e9fb076f83c4c8c2692\": container with ID starting with 6b05cb463042a2d756c62db149db9a1150af1e92d97a5e9fb076f83c4c8c2692 not found: ID does not exist"
Sep 12 01:11:01.724595 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.724570    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"]
Sep 12 01:11:01.728409 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.728387    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-61f73f/auth-test-model-61f73f-kserve-684cd59855-dwf8l"]
Sep 12 01:11:01.790721 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.790685    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-tls-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:11:01.790721 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.790720    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-dshm\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:11:01.790853 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.790733    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-home\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:11:01.790853 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:01.790742    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnfjm\" (UniqueName: \"kubernetes.io/projected/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e-kube-api-access-dnfjm\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:11:03.590460 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:11:03.590426    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf9f0c29-9c94-47c2-bc4e-09ad14d3041e" path="/var/lib/kubelet/pods/bf9f0c29-9c94-47c2-bc4e-09ad14d3041e/volumes"
Sep 12 01:12:55.643924 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:55.643893    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 01:12:55.645626 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:55.645584    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 01:12:59.826424 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826391    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"]
Sep 12 01:12:59.826955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826734    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d0c2aa1-7e24-4d5b-a711-1f8657dbe378" containerName="payload-processing"
Sep 12 01:12:59.826955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826747    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d0c2aa1-7e24-4d5b-a711-1f8657dbe378" containerName="payload-processing"
Sep 12 01:12:59.826955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826757    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" containerName="istio-proxy"
Sep 12 01:12:59.826955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826763    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" containerName="istio-proxy"
Sep 12 01:12:59.826955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826770    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e26c7769-35d9-4d19-9f99-273f4193b16c" containerName="payload-processing"
Sep 12 01:12:59.826955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826775    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e26c7769-35d9-4d19-9f99-273f4193b16c" containerName="payload-processing"
Sep 12 01:12:59.826955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826782    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="606bd754-eb44-4cb7-85c5-b68ba4115dd1" containerName="payload-pre-processing"
Sep 12 01:12:59.826955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826787    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="606bd754-eb44-4cb7-85c5-b68ba4115dd1" containerName="payload-pre-processing"
Sep 12 01:12:59.826955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826797    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" containerName="istio-proxy"
Sep 12 01:12:59.826955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826805    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" containerName="istio-proxy"
Sep 12 01:12:59.826955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826859    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f222790-5b43-4c49-b428-400ebf670c1f" containerName="revoke-keys"
Sep 12 01:12:59.826955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826871    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f222790-5b43-4c49-b428-400ebf670c1f" containerName="revoke-keys"
Sep 12 01:12:59.826955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826891    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="171640c8-513b-4b58-a865-08d1cf2faf75" containerName="istio-proxy"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826968    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="171640c8-513b-4b58-a865-08d1cf2faf75" containerName="istio-proxy"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.826994    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="171640c8-513b-4b58-a865-08d1cf2faf75" containerName="istio-proxy"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.827004    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="171640c8-513b-4b58-a865-08d1cf2faf75" containerName="istio-proxy"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.827020    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d98b6109-80a7-4ad8-8795-47430bea0a98" containerName="payload-pre-processing"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.827027    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d98b6109-80a7-4ad8-8795-47430bea0a98" containerName="payload-pre-processing"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.827039    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf9f0c29-9c94-47c2-bc4e-09ad14d3041e" containerName="main"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.827047    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf9f0c29-9c94-47c2-bc4e-09ad14d3041e" containerName="main"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.827147    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d0c2aa1-7e24-4d5b-a711-1f8657dbe378" containerName="payload-processing"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.827163    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" containerName="istio-proxy"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.827173    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf9f0c29-9c94-47c2-bc4e-09ad14d3041e" containerName="main"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.827181    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="606bd754-eb44-4cb7-85c5-b68ba4115dd1" containerName="payload-pre-processing"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.827191    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e26c7769-35d9-4d19-9f99-273f4193b16c" containerName="payload-processing"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.827200    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="171640c8-513b-4b58-a865-08d1cf2faf75" containerName="istio-proxy"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.827209    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f222790-5b43-4c49-b428-400ebf670c1f" containerName="revoke-keys"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.827219    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d98b6109-80a7-4ad8-8795-47430bea0a98" containerName="payload-pre-processing"
Sep 12 01:12:59.827696 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.827228    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="171640c8-513b-4b58-a865-08d1cf2faf75" containerName="istio-proxy"
Sep 12 01:12:59.830072 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.830048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:12:59.832787 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.832763    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ef5997\"/\"kube-root-ca.crt\""
Sep 12 01:12:59.833705 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.833686    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ef5997\"/\"openshift-service-ca.crt\""
Sep 12 01:12:59.833843 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.833803    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ef5997\"/\"default-dockercfg-fh9nk\""
Sep 12 01:12:59.833922 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.833878    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ef5997\"/\"rate-test-model-ef5997-kserve-self-signed-certs\""
Sep 12 01:12:59.846836 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.846812    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"]
Sep 12 01:12:59.934312 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.934279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94rb9\" (UniqueName: \"kubernetes.io/projected/88459b8f-f276-48aa-8f7e-dc74c7a0a227-kube-api-access-94rb9\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:12:59.934462 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.934346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-dshm\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:12:59.934462 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.934424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-tmp-dir\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:12:59.934556 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.934468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-home\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:12:59.934556 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.934486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88459b8f-f276-48aa-8f7e-dc74c7a0a227-tls-certs\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:12:59.934556 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:12:59.934533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-model-cache\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:00.035230 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:00.035160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-tmp-dir\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:00.035457 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:00.035257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-home\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:00.035457 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:00.035286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88459b8f-f276-48aa-8f7e-dc74c7a0a227-tls-certs\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:00.035457 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:00.035323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-model-cache\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:00.035457 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:00.035360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94rb9\" (UniqueName: \"kubernetes.io/projected/88459b8f-f276-48aa-8f7e-dc74c7a0a227-kube-api-access-94rb9\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:00.035457 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:00.035424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-dshm\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:00.035764 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:00.035694    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-model-cache\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:00.035913 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:00.035887    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-home\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:00.036180 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:00.036158    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-tmp-dir\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:00.038518 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:00.038496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-dshm\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:00.038690 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:00.038528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88459b8f-f276-48aa-8f7e-dc74c7a0a227-tls-certs\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:00.045110 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:00.045074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94rb9\" (UniqueName: \"kubernetes.io/projected/88459b8f-f276-48aa-8f7e-dc74c7a0a227-kube-api-access-94rb9\") pod \"rate-test-model-ef5997-kserve-cf795d9f9-ghwx2\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") " pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:00.146774 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:00.146704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:00.416741 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:00.416713    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"]
Sep 12 01:13:00.419702 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:13:00.419676    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88459b8f_f276_48aa_8f7e_dc74c7a0a227.slice/crio-da38aaaad5448b2c2bed3158ecd23c9b9431b115eb103198c6dbb09b84f7fafc WatchSource:0}: Error finding container da38aaaad5448b2c2bed3158ecd23c9b9431b115eb103198c6dbb09b84f7fafc: Status 404 returned error can't find the container with id da38aaaad5448b2c2bed3158ecd23c9b9431b115eb103198c6dbb09b84f7fafc
Sep 12 01:13:01.252924 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:01.252886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2" event={"ID":"88459b8f-f276-48aa-8f7e-dc74c7a0a227","Type":"ContainerStarted","Data":"a67173da22e0e1e6d5c0ed50af51244f321b1f872504c3392d3489beea0e633f"}
Sep 12 01:13:01.252924 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:01.252930    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2" event={"ID":"88459b8f-f276-48aa-8f7e-dc74c7a0a227","Type":"ContainerStarted","Data":"da38aaaad5448b2c2bed3158ecd23c9b9431b115eb103198c6dbb09b84f7fafc"}
Sep 12 01:13:01.271584 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:01.271535    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2" podStartSLOduration=2.271519346 podStartE2EDuration="2.271519346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:13:01.271185635 +0000 UTC m=+1806.190477422" watchObservedRunningTime="2026-09-12 01:13:01.271519346 +0000 UTC m=+1806.190811134"
Sep 12 01:13:10.147036 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:10.146995    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:10.147036 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:10.147039    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:10.159319 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:10.159295    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:10.310173 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:10.310139    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:34.318995 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:34.318961    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"]
Sep 12 01:13:35.083461 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.083427    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"]
Sep 12 01:13:35.083875 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.083856    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0b453e-3223-4a85-956d-d868ea6e4e9f" containerName="istio-proxy"
Sep 12 01:13:35.087094 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.087073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.089726 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.089705    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5fbbcd-openshift-default-dockercfg-4pzdp\""
Sep 12 01:13:35.101030 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.101008    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"]
Sep 12 01:13:35.217525 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.217486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-credential-socket\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.217719 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.217538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17426b08-e769-4573-897e-14736c23767f-istio-token\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.217719 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.217612    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nhbn\" (UniqueName: \"kubernetes.io/projected/17426b08-e769-4573-897e-14736c23767f-kube-api-access-9nhbn\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.217719 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.217660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-istio-envoy\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.217719 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.217686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17426b08-e769-4573-897e-14736c23767f-istiod-ca-cert\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.217888 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.217730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-workload-certs\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.217888 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.217767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-istio-data\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.217888 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.217784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17426b08-e769-4573-897e-14736c23767f-istio-podinfo\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.217888 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.217827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-workload-socket\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.318928 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.318899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-credential-socket\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.319120 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.318936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17426b08-e769-4573-897e-14736c23767f-istio-token\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.319120 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.318953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nhbn\" (UniqueName: \"kubernetes.io/projected/17426b08-e769-4573-897e-14736c23767f-kube-api-access-9nhbn\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.319120 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.319074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-istio-envoy\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.319630 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.319121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17426b08-e769-4573-897e-14736c23767f-istiod-ca-cert\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.319630 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.319240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-workload-certs\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.319630 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.319296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-istio-data\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.319630 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.319324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17426b08-e769-4573-897e-14736c23767f-istio-podinfo\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.319630 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.319352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-credential-socket\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.319630 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.319391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-workload-socket\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.319949 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.319642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-istio-data\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.319949 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.319696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-workload-socket\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.319949 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.319794    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17426b08-e769-4573-897e-14736c23767f-istiod-ca-cert\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.319949 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.319846    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-workload-certs\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.321393 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.321361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-istio-envoy\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.321735 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.321716    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17426b08-e769-4573-897e-14736c23767f-istio-podinfo\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.327836 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.327789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17426b08-e769-4573-897e-14736c23767f-istio-token\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.328248 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.328225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nhbn\" (UniqueName: \"kubernetes.io/projected/17426b08-e769-4573-897e-14736c23767f-kube-api-access-9nhbn\") pod \"e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") " pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.397329 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.397264    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:35.675144 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.675117    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"]
Sep 12 01:13:35.677208 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:13:35.677182    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17426b08_e769_4573_897e_14736c23767f.slice/crio-c8e2a60c4c15dbc6435f77b086e01f2f1a733410310a44cf6344c5e45921fa35 WatchSource:0}: Error finding container c8e2a60c4c15dbc6435f77b086e01f2f1a733410310a44cf6344c5e45921fa35: Status 404 returned error can't find the container with id c8e2a60c4c15dbc6435f77b086e01f2f1a733410310a44cf6344c5e45921fa35
Sep 12 01:13:35.679421 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.679391    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 01:13:35.679513 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.679468    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 01:13:35.679575 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:35.679512    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 01:13:36.424276 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:36.424242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl" event={"ID":"17426b08-e769-4573-897e-14736c23767f","Type":"ContainerStarted","Data":"26a1e0920a415cfd58f3b304211b834f138e4512ed9388eadd816e636e22131f"}
Sep 12 01:13:36.424775 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:36.424297    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl" event={"ID":"17426b08-e769-4573-897e-14736c23767f","Type":"ContainerStarted","Data":"c8e2a60c4c15dbc6435f77b086e01f2f1a733410310a44cf6344c5e45921fa35"}
Sep 12 01:13:36.445849 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:36.445803    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl" podStartSLOduration=1.445782489 podStartE2EDuration="1.445782489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:13:36.444268312 +0000 UTC m=+1841.363560110" watchObservedRunningTime="2026-09-12 01:13:36.445782489 +0000 UTC m=+1841.365074328"
Sep 12 01:13:37.397812 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:37.397772    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:37.404714 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:37.404672    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:37.429427 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:37.429395    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:37.430291 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:37.430272    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:40.454771 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.454736    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"]
Sep 12 01:13:40.458317 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.458294    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"
Sep 12 01:13:40.460913 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.460887    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5fbbcd\""
Sep 12 01:13:40.461007 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.460997    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5fbbcd-dockercfg-dfmqw\""
Sep 12 01:13:40.471325 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.471301    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"]
Sep 12 01:13:40.474269 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.474254    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"
Sep 12 01:13:40.479476 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.479452    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"]
Sep 12 01:13:40.500518 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.500493    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"]
Sep 12 01:13:40.567360 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.567319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5vsz\" (UniqueName: \"kubernetes.io/projected/bd89d965-fd17-4da6-b568-68b2fd2432e0-kube-api-access-p5vsz\") pod \"payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg\" (UID: \"bd89d965-fd17-4da6-b568-68b2fd2432e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"
Sep 12 01:13:40.567360 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.567371    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5b2r\" (UniqueName: \"kubernetes.io/projected/50ff1acc-4e2a-487d-a907-aea16d94f938-kube-api-access-j5b2r\") pod \"payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf\" (UID: \"50ff1acc-4e2a-487d-a907-aea16d94f938\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"
Sep 12 01:13:40.567651 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.567415    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ff1acc-4e2a-487d-a907-aea16d94f938-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf\" (UID: \"50ff1acc-4e2a-487d-a907-aea16d94f938\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"
Sep 12 01:13:40.567651 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.567460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd89d965-fd17-4da6-b568-68b2fd2432e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg\" (UID: \"bd89d965-fd17-4da6-b568-68b2fd2432e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"
Sep 12 01:13:40.668302 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.668265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5vsz\" (UniqueName: \"kubernetes.io/projected/bd89d965-fd17-4da6-b568-68b2fd2432e0-kube-api-access-p5vsz\") pod \"payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg\" (UID: \"bd89d965-fd17-4da6-b568-68b2fd2432e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"
Sep 12 01:13:40.668302 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.668305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5b2r\" (UniqueName: \"kubernetes.io/projected/50ff1acc-4e2a-487d-a907-aea16d94f938-kube-api-access-j5b2r\") pod \"payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf\" (UID: \"50ff1acc-4e2a-487d-a907-aea16d94f938\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"
Sep 12 01:13:40.668524 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.668332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ff1acc-4e2a-487d-a907-aea16d94f938-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf\" (UID: \"50ff1acc-4e2a-487d-a907-aea16d94f938\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"
Sep 12 01:13:40.668524 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.668366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd89d965-fd17-4da6-b568-68b2fd2432e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg\" (UID: \"bd89d965-fd17-4da6-b568-68b2fd2432e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"
Sep 12 01:13:40.669317 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.669290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ff1acc-4e2a-487d-a907-aea16d94f938-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf\" (UID: \"50ff1acc-4e2a-487d-a907-aea16d94f938\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"
Sep 12 01:13:40.669401 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.669290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd89d965-fd17-4da6-b568-68b2fd2432e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg\" (UID: \"bd89d965-fd17-4da6-b568-68b2fd2432e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"
Sep 12 01:13:40.677152 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.677125    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5b2r\" (UniqueName: \"kubernetes.io/projected/50ff1acc-4e2a-487d-a907-aea16d94f938-kube-api-access-j5b2r\") pod \"payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf\" (UID: \"50ff1acc-4e2a-487d-a907-aea16d94f938\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"
Sep 12 01:13:40.677269 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.677182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5vsz\" (UniqueName: \"kubernetes.io/projected/bd89d965-fd17-4da6-b568-68b2fd2432e0-kube-api-access-p5vsz\") pod \"payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg\" (UID: \"bd89d965-fd17-4da6-b568-68b2fd2432e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"
Sep 12 01:13:40.769083 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.769005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"
Sep 12 01:13:40.784945 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:40.784919    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"
Sep 12 01:13:41.045629 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:41.045590    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"]
Sep 12 01:13:41.047813 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:13:41.047782    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd89d965_fd17_4da6_b568_68b2fd2432e0.slice/crio-29551895be8ceaaf108eaf0f67b55a634e71601d26d611e5fe968a19e0eeeaec WatchSource:0}: Error finding container 29551895be8ceaaf108eaf0f67b55a634e71601d26d611e5fe968a19e0eeeaec: Status 404 returned error can't find the container with id 29551895be8ceaaf108eaf0f67b55a634e71601d26d611e5fe968a19e0eeeaec
Sep 12 01:13:41.073538 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:41.073515    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"]
Sep 12 01:13:41.077654 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:13:41.077626    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ff1acc_4e2a_487d_a907_aea16d94f938.slice/crio-04fdcba54e2aff34c9da14cc0e5ca3f58852fdfc5a503b06e65af781c9ce65f2 WatchSource:0}: Error finding container 04fdcba54e2aff34c9da14cc0e5ca3f58852fdfc5a503b06e65af781c9ce65f2: Status 404 returned error can't find the container with id 04fdcba54e2aff34c9da14cc0e5ca3f58852fdfc5a503b06e65af781c9ce65f2
Sep 12 01:13:41.449698 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:41.449661    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf" event={"ID":"50ff1acc-4e2a-487d-a907-aea16d94f938","Type":"ContainerStarted","Data":"8a4ec2bb70427942d0a13d71981121847ae2e5dcccf5a7922d77f28fd3d6532b"}
Sep 12 01:13:41.449901 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:41.449718    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf" event={"ID":"50ff1acc-4e2a-487d-a907-aea16d94f938","Type":"ContainerStarted","Data":"04fdcba54e2aff34c9da14cc0e5ca3f58852fdfc5a503b06e65af781c9ce65f2"}
Sep 12 01:13:41.449901 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:41.449743    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"
Sep 12 01:13:41.451245 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:41.451223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg" event={"ID":"bd89d965-fd17-4da6-b568-68b2fd2432e0","Type":"ContainerStarted","Data":"7815951057de79acf88a711e8a71f4161ad53599a349a9c64906d0324e600b9f"}
Sep 12 01:13:41.451320 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:41.451266    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg" event={"ID":"bd89d965-fd17-4da6-b568-68b2fd2432e0","Type":"ContainerStarted","Data":"29551895be8ceaaf108eaf0f67b55a634e71601d26d611e5fe968a19e0eeeaec"}
Sep 12 01:13:41.451383 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:41.451332    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"
Sep 12 01:13:41.467463 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:41.467417    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf" podStartSLOduration=1.467402688 podStartE2EDuration="1.467402688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:13:41.465913678 +0000 UTC m=+1846.385205480" watchObservedRunningTime="2026-09-12 01:13:41.467402688 +0000 UTC m=+1846.386694483"
Sep 12 01:13:41.485723 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:41.485681    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg" podStartSLOduration=1.485667794 podStartE2EDuration="1.485667794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:13:41.484318013 +0000 UTC m=+1846.403609813" watchObservedRunningTime="2026-09-12 01:13:41.485667794 +0000 UTC m=+1846.404959582"
Sep 12 01:13:49.337931 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.337872    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2" podUID="88459b8f-f276-48aa-8f7e-dc74c7a0a227" containerName="main" containerID="cri-o://a67173da22e0e1e6d5c0ed50af51244f321b1f872504c3392d3489beea0e633f" gracePeriod=45
Sep 12 01:13:49.496757 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.496726    2579 generic.go:356] "Generic (PLEG): container finished" podID="88459b8f-f276-48aa-8f7e-dc74c7a0a227" containerID="a67173da22e0e1e6d5c0ed50af51244f321b1f872504c3392d3489beea0e633f" exitCode=0
Sep 12 01:13:49.496905 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.496796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2" event={"ID":"88459b8f-f276-48aa-8f7e-dc74c7a0a227","Type":"ContainerDied","Data":"a67173da22e0e1e6d5c0ed50af51244f321b1f872504c3392d3489beea0e633f"}
Sep 12 01:13:49.616385 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.616298    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:49.749890 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.749854    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94rb9\" (UniqueName: \"kubernetes.io/projected/88459b8f-f276-48aa-8f7e-dc74c7a0a227-kube-api-access-94rb9\") pod \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") "
Sep 12 01:13:49.750087 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.749901    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-tmp-dir\") pod \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") "
Sep 12 01:13:49.750087 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.749925    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-model-cache\") pod \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") "
Sep 12 01:13:49.750087 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.749948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-dshm\") pod \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") "
Sep 12 01:13:49.750087 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.749963    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88459b8f-f276-48aa-8f7e-dc74c7a0a227-tls-certs\") pod \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") "
Sep 12 01:13:49.750087 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.750018    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-home\") pod \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\" (UID: \"88459b8f-f276-48aa-8f7e-dc74c7a0a227\") "
Sep 12 01:13:49.750376 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.750167    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88459b8f-f276-48aa-8f7e-dc74c7a0a227" (UID: "88459b8f-f276-48aa-8f7e-dc74c7a0a227"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:13:49.750376 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.750292    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-home" (OuterVolumeSpecName: "home") pod "88459b8f-f276-48aa-8f7e-dc74c7a0a227" (UID: "88459b8f-f276-48aa-8f7e-dc74c7a0a227"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:13:49.750376 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.750317    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-tmp-dir\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:13:49.750376 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.750366    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-model-cache" (OuterVolumeSpecName: "model-cache") pod "88459b8f-f276-48aa-8f7e-dc74c7a0a227" (UID: "88459b8f-f276-48aa-8f7e-dc74c7a0a227"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:13:49.752145 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.752120    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88459b8f-f276-48aa-8f7e-dc74c7a0a227-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88459b8f-f276-48aa-8f7e-dc74c7a0a227" (UID: "88459b8f-f276-48aa-8f7e-dc74c7a0a227"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:13:49.752438 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.752415    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-dshm" (OuterVolumeSpecName: "dshm") pod "88459b8f-f276-48aa-8f7e-dc74c7a0a227" (UID: "88459b8f-f276-48aa-8f7e-dc74c7a0a227"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:13:49.752482 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.752415    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88459b8f-f276-48aa-8f7e-dc74c7a0a227-kube-api-access-94rb9" (OuterVolumeSpecName: "kube-api-access-94rb9") pod "88459b8f-f276-48aa-8f7e-dc74c7a0a227" (UID: "88459b8f-f276-48aa-8f7e-dc74c7a0a227"). InnerVolumeSpecName "kube-api-access-94rb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:13:49.850919 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.850878    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-home\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:13:49.850919 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.850916    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94rb9\" (UniqueName: \"kubernetes.io/projected/88459b8f-f276-48aa-8f7e-dc74c7a0a227-kube-api-access-94rb9\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:13:49.850919 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.850928    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-model-cache\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:13:49.851141 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.850937    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88459b8f-f276-48aa-8f7e-dc74c7a0a227-dshm\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:13:49.851141 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:49.850945    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88459b8f-f276-48aa-8f7e-dc74c7a0a227-tls-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:13:50.506591 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:50.506546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2" event={"ID":"88459b8f-f276-48aa-8f7e-dc74c7a0a227","Type":"ContainerDied","Data":"da38aaaad5448b2c2bed3158ecd23c9b9431b115eb103198c6dbb09b84f7fafc"}
Sep 12 01:13:50.507058 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:50.506638    2579 scope.go:117] "RemoveContainer" containerID="a67173da22e0e1e6d5c0ed50af51244f321b1f872504c3392d3489beea0e633f"
Sep 12 01:13:50.507058 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:50.506849    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"
Sep 12 01:13:50.539539 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:50.539511    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"]
Sep 12 01:13:50.543496 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:50.543468    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ef5997/rate-test-model-ef5997-kserve-cf795d9f9-ghwx2"]
Sep 12 01:13:51.591533 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:51.591501    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88459b8f-f276-48aa-8f7e-dc74c7a0a227" path="/var/lib/kubelet/pods/88459b8f-f276-48aa-8f7e-dc74c7a0a227/volumes"
Sep 12 01:13:52.463530 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:52.463500    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"
Sep 12 01:13:52.463752 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:52.463581    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"
Sep 12 01:13:53.470752 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:53.470713    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 12 01:13:53.471116 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:53.470777    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl" podUID="17426b08-e769-4573-897e-14736c23767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 12 01:13:54.533157 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:54.533128    2579 generic.go:356] "Generic (PLEG): container finished" podID="17426b08-e769-4573-897e-14736c23767f" containerID="26a1e0920a415cfd58f3b304211b834f138e4512ed9388eadd816e636e22131f" exitCode=0
Sep 12 01:13:54.533569 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:54.533205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl" event={"ID":"17426b08-e769-4573-897e-14736c23767f","Type":"ContainerDied","Data":"26a1e0920a415cfd58f3b304211b834f138e4512ed9388eadd816e636e22131f"}
Sep 12 01:13:54.533651 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:54.533637    2579 scope.go:117] "RemoveContainer" containerID="26a1e0920a415cfd58f3b304211b834f138e4512ed9388eadd816e636e22131f"
Sep 12 01:13:54.534677 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:54.534655    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 01:13:54.534748 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:54.534716    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 01:13:54.534785 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:54.534746    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 12 01:13:55.540107 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:55.540075    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl" event={"ID":"17426b08-e769-4573-897e-14736c23767f","Type":"ContainerStarted","Data":"00e115b9870a179d5e3f1b1a40d8e0cf1e1ed0251b0fd2598c57c1aba193d87b"}
Sep 12 01:13:56.397704 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:56.397645    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:13:56.397927 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:56.397898    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 12 01:13:56.398009 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:56.397959    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl" podUID="17426b08-e769-4573-897e-14736c23767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 12 01:13:57.398829 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:57.398790    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 12 01:13:57.399418 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:57.399385    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl" podUID="17426b08-e769-4573-897e-14736c23767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 12 01:13:58.398014 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:58.397980    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 12 01:13:58.398347 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:58.398307    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl" podUID="17426b08-e769-4573-897e-14736c23767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 12 01:13:59.398593 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:59.398553    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 12 01:13:59.399231 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:13:59.399198    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl" podUID="17426b08-e769-4573-897e-14736c23767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 12 01:14:00.398288 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:00.398248    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Sep 12 01:14:00.398597 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:00.398566    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl" podUID="17426b08-e769-4573-897e-14736c23767f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Sep 12 01:14:01.495408 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:01.495377    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:14:01.495891 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:01.495636    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:14:01.496575 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:01.496556    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:14:10.932176 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:10.932142    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"]
Sep 12 01:14:10.932800 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:10.932671    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88459b8f-f276-48aa-8f7e-dc74c7a0a227" containerName="main"
Sep 12 01:14:10.932800 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:10.932693    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="88459b8f-f276-48aa-8f7e-dc74c7a0a227" containerName="main"
Sep 12 01:14:10.932800 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:10.932790    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="88459b8f-f276-48aa-8f7e-dc74c7a0a227" containerName="main"
Sep 12 01:14:10.936004 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:10.935985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"
Sep 12 01:14:10.938636 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:10.938589    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-57b051\""
Sep 12 01:14:10.938725 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:10.938653    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-57b051-dockercfg-z5682\""
Sep 12 01:14:10.947793 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:10.947770    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"]
Sep 12 01:14:10.949891 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:10.949872    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"]
Sep 12 01:14:10.953043 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:10.953023    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"
Sep 12 01:14:10.965955 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:10.965937    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"]
Sep 12 01:14:11.037791 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.037749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90f67e45-b5e8-4ef5-b796-c4a0f26c207a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp\" (UID: \"90f67e45-b5e8-4ef5-b796-c4a0f26c207a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"
Sep 12 01:14:11.037984 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.037815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpp8l\" (UniqueName: \"kubernetes.io/projected/90f67e45-b5e8-4ef5-b796-c4a0f26c207a-kube-api-access-gpp8l\") pod \"payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp\" (UID: \"90f67e45-b5e8-4ef5-b796-c4a0f26c207a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"
Sep 12 01:14:11.037984 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.037912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c239a42-fbfa-42d7-a670-3bf0c77a8c80-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth\" (UID: \"8c239a42-fbfa-42d7-a670-3bf0c77a8c80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"
Sep 12 01:14:11.038119 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.037986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx9qg\" (UniqueName: \"kubernetes.io/projected/8c239a42-fbfa-42d7-a670-3bf0c77a8c80-kube-api-access-qx9qg\") pod \"payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth\" (UID: \"8c239a42-fbfa-42d7-a670-3bf0c77a8c80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"
Sep 12 01:14:11.138921 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.138884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx9qg\" (UniqueName: \"kubernetes.io/projected/8c239a42-fbfa-42d7-a670-3bf0c77a8c80-kube-api-access-qx9qg\") pod \"payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth\" (UID: \"8c239a42-fbfa-42d7-a670-3bf0c77a8c80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"
Sep 12 01:14:11.139089 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.138955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90f67e45-b5e8-4ef5-b796-c4a0f26c207a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp\" (UID: \"90f67e45-b5e8-4ef5-b796-c4a0f26c207a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"
Sep 12 01:14:11.139089 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.139001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpp8l\" (UniqueName: \"kubernetes.io/projected/90f67e45-b5e8-4ef5-b796-c4a0f26c207a-kube-api-access-gpp8l\") pod \"payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp\" (UID: \"90f67e45-b5e8-4ef5-b796-c4a0f26c207a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"
Sep 12 01:14:11.139089 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.139050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c239a42-fbfa-42d7-a670-3bf0c77a8c80-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth\" (UID: \"8c239a42-fbfa-42d7-a670-3bf0c77a8c80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"
Sep 12 01:14:11.139744 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.139724    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c239a42-fbfa-42d7-a670-3bf0c77a8c80-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth\" (UID: \"8c239a42-fbfa-42d7-a670-3bf0c77a8c80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"
Sep 12 01:14:11.139900 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.139875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90f67e45-b5e8-4ef5-b796-c4a0f26c207a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp\" (UID: \"90f67e45-b5e8-4ef5-b796-c4a0f26c207a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"
Sep 12 01:14:11.147287 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.147251    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx9qg\" (UniqueName: \"kubernetes.io/projected/8c239a42-fbfa-42d7-a670-3bf0c77a8c80-kube-api-access-qx9qg\") pod \"payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth\" (UID: \"8c239a42-fbfa-42d7-a670-3bf0c77a8c80\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"
Sep 12 01:14:11.147738 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.147716    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpp8l\" (UniqueName: \"kubernetes.io/projected/90f67e45-b5e8-4ef5-b796-c4a0f26c207a-kube-api-access-gpp8l\") pod \"payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp\" (UID: \"90f67e45-b5e8-4ef5-b796-c4a0f26c207a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"
Sep 12 01:14:11.246962 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.246897    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"
Sep 12 01:14:11.262968 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.262945    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"
Sep 12 01:14:11.525888 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.525792    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"]
Sep 12 01:14:11.529423 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:14:11.529392    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f67e45_b5e8_4ef5_b796_c4a0f26c207a.slice/crio-c65eb2de30484a3f98b029db7c6bf9d6471e0be70da025a6b0c700d47d83eea9 WatchSource:0}: Error finding container c65eb2de30484a3f98b029db7c6bf9d6471e0be70da025a6b0c700d47d83eea9: Status 404 returned error can't find the container with id c65eb2de30484a3f98b029db7c6bf9d6471e0be70da025a6b0c700d47d83eea9
Sep 12 01:14:11.546806 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.546786    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"]
Sep 12 01:14:11.548679 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:14:11.548635    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c239a42_fbfa_42d7_a670_3bf0c77a8c80.slice/crio-69d2c104b2f3ed98917f9c248c0b4fc4c231127c514ce631f90bf2ba59df9f15 WatchSource:0}: Error finding container 69d2c104b2f3ed98917f9c248c0b4fc4c231127c514ce631f90bf2ba59df9f15: Status 404 returned error can't find the container with id 69d2c104b2f3ed98917f9c248c0b4fc4c231127c514ce631f90bf2ba59df9f15
Sep 12 01:14:11.625467 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.625363    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp" event={"ID":"90f67e45-b5e8-4ef5-b796-c4a0f26c207a","Type":"ContainerStarted","Data":"9199580877a9439d4622dccf2ae3a3027fb67f45e7a313b867a8e522af6651b0"}
Sep 12 01:14:11.625467 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.625431    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp" event={"ID":"90f67e45-b5e8-4ef5-b796-c4a0f26c207a","Type":"ContainerStarted","Data":"c65eb2de30484a3f98b029db7c6bf9d6471e0be70da025a6b0c700d47d83eea9"}
Sep 12 01:14:11.625995 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.625973    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"
Sep 12 01:14:11.627975 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.627946    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth" event={"ID":"8c239a42-fbfa-42d7-a670-3bf0c77a8c80","Type":"ContainerStarted","Data":"3409e665ee3f2726cbcbd04267373bd8278ffc24afbbee852e78f3635ddf38bd"}
Sep 12 01:14:11.628082 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.627998    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth" event={"ID":"8c239a42-fbfa-42d7-a670-3bf0c77a8c80","Type":"ContainerStarted","Data":"69d2c104b2f3ed98917f9c248c0b4fc4c231127c514ce631f90bf2ba59df9f15"}
Sep 12 01:14:11.628369 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.628350    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"
Sep 12 01:14:11.645408 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.645295    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp" podStartSLOduration=1.645276742 podStartE2EDuration="1.645276742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:14:11.645151655 +0000 UTC m=+1876.564443468" watchObservedRunningTime="2026-09-12 01:14:11.645276742 +0000 UTC m=+1876.564568549"
Sep 12 01:14:11.664258 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:11.664217    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth" podStartSLOduration=1.664204843 podStartE2EDuration="1.664204843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:14:11.663286466 +0000 UTC m=+1876.582578255" watchObservedRunningTime="2026-09-12 01:14:11.664204843 +0000 UTC m=+1876.583496626"
Sep 12 01:14:22.635372 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:22.635298    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"
Sep 12 01:14:23.640079 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:23.640050    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"
Sep 12 01:14:37.935247 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:37.935215    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-plxcj_f4fff3f1-7b21-4c2d-a4fc-33d2a733834e/payload-processing/0.log"
Sep 12 01:14:38.059298 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:14:38.059272    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth_8c239a42-fbfa-42d7-a670-3bf0c77a8c80/payload-processing/0.log"
Sep 12 01:15:00.136508 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.136476    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819595-zsqp4"]
Sep 12 01:15:00.139769 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.139753    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-57b051-29819595-tsb6g"]
Sep 12 01:15:00.139881 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.139861    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819595-zsqp4"
Sep 12 01:15:00.142502 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.142480    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2rn9k\""
Sep 12 01:15:00.143013 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.142992    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-57b051-29819595-tsb6g"
Sep 12 01:15:00.149141 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.149121    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819595-zsqp4"]
Sep 12 01:15:00.151486 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.151461    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-57b051-29819595-tsb6g"]
Sep 12 01:15:00.152491 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.152473    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819595-zsqp4"
Sep 12 01:15:00.160798 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.160775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-57b051-29819595-tsb6g"
Sep 12 01:15:00.431909 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.431885    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819595-zsqp4"]
Sep 12 01:15:00.433882 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:15:00.433850    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ae4a30_e750_45d0_8b72_93ef713187b7.slice/crio-6ca465b7fd7c905cea365f65b3112a828821749327cdfb85afc55bb5c8cf7f64 WatchSource:0}: Error finding container 6ca465b7fd7c905cea365f65b3112a828821749327cdfb85afc55bb5c8cf7f64: Status 404 returned error can't find the container with id 6ca465b7fd7c905cea365f65b3112a828821749327cdfb85afc55bb5c8cf7f64
Sep 12 01:15:00.440440 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.440419    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-57b051-29819595-tsb6g"]
Sep 12 01:15:00.442708 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:15:00.442680    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06bf9a2_8993_4311_9d28_635593b6ae84.slice/crio-d4ef62303112ae4c2c04b3e954aef3586ebb7ce64ad87e83ee8bfe1cd04ce8dc WatchSource:0}: Error finding container d4ef62303112ae4c2c04b3e954aef3586ebb7ce64ad87e83ee8bfe1cd04ce8dc: Status 404 returned error can't find the container with id d4ef62303112ae4c2c04b3e954aef3586ebb7ce64ad87e83ee8bfe1cd04ce8dc
Sep 12 01:15:00.877045 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.876999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-57b051-29819595-tsb6g" event={"ID":"d06bf9a2-8993-4311-9d28-635593b6ae84","Type":"ContainerStarted","Data":"40b415c059e9fd0f79f73b69750c4c2da8a0a23c1a18a354969917415f189113"}
Sep 12 01:15:00.877220 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.877093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-57b051-29819595-tsb6g" event={"ID":"d06bf9a2-8993-4311-9d28-635593b6ae84","Type":"ContainerStarted","Data":"d4ef62303112ae4c2c04b3e954aef3586ebb7ce64ad87e83ee8bfe1cd04ce8dc"}
Sep 12 01:15:00.878732 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.878709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819595-zsqp4" event={"ID":"a6ae4a30-e750-45d0-8b72-93ef713187b7","Type":"ContainerStarted","Data":"6c8296e8e9be43de4102e2b606fe69066e2ce0f6092fe5131ab9293537b21dab"}
Sep 12 01:15:00.878836 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.878752    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819595-zsqp4" event={"ID":"a6ae4a30-e750-45d0-8b72-93ef713187b7","Type":"ContainerStarted","Data":"6ca465b7fd7c905cea365f65b3112a828821749327cdfb85afc55bb5c8cf7f64"}
Sep 12 01:15:00.893309 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.893261    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-57b051-29819595-tsb6g" podStartSLOduration=0.893245629 podStartE2EDuration="893.245629ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:15:00.892271176 +0000 UTC m=+1925.811562963" watchObservedRunningTime="2026-09-12 01:15:00.893245629 +0000 UTC m=+1925.812537415"
Sep 12 01:15:00.906927 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:00.906890    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819595-zsqp4" podStartSLOduration=0.906877744 podStartE2EDuration="906.877744ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:15:00.906322502 +0000 UTC m=+1925.825614290" watchObservedRunningTime="2026-09-12 01:15:00.906877744 +0000 UTC m=+1925.826169530"
Sep 12 01:15:01.885544 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:01.885505    2579 generic.go:356] "Generic (PLEG): container finished" podID="d06bf9a2-8993-4311-9d28-635593b6ae84" containerID="40b415c059e9fd0f79f73b69750c4c2da8a0a23c1a18a354969917415f189113" exitCode=0
Sep 12 01:15:01.885945 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:01.885567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-57b051-29819595-tsb6g" event={"ID":"d06bf9a2-8993-4311-9d28-635593b6ae84","Type":"ContainerDied","Data":"40b415c059e9fd0f79f73b69750c4c2da8a0a23c1a18a354969917415f189113"}
Sep 12 01:15:01.887141 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:01.887119    2579 generic.go:356] "Generic (PLEG): container finished" podID="a6ae4a30-e750-45d0-8b72-93ef713187b7" containerID="6c8296e8e9be43de4102e2b606fe69066e2ce0f6092fe5131ab9293537b21dab" exitCode=0
Sep 12 01:15:01.887242 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:01.887149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819595-zsqp4" event={"ID":"a6ae4a30-e750-45d0-8b72-93ef713187b7","Type":"ContainerDied","Data":"6c8296e8e9be43de4102e2b606fe69066e2ce0f6092fe5131ab9293537b21dab"}
Sep 12 01:15:03.022214 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:03.022186    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819595-zsqp4"
Sep 12 01:15:03.060895 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:03.060872    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-57b051-29819595-tsb6g"
Sep 12 01:15:03.898544 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:03.898460    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819595-zsqp4"
Sep 12 01:15:03.898544 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:03.898497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819595-zsqp4" event={"ID":"a6ae4a30-e750-45d0-8b72-93ef713187b7","Type":"ContainerDied","Data":"6ca465b7fd7c905cea365f65b3112a828821749327cdfb85afc55bb5c8cf7f64"}
Sep 12 01:15:03.898544 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:03.898540    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ca465b7fd7c905cea365f65b3112a828821749327cdfb85afc55bb5c8cf7f64"
Sep 12 01:15:03.899908 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:03.899879    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-57b051-29819595-tsb6g"
Sep 12 01:15:03.900043 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:03.899875    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-57b051-29819595-tsb6g" event={"ID":"d06bf9a2-8993-4311-9d28-635593b6ae84","Type":"ContainerDied","Data":"d4ef62303112ae4c2c04b3e954aef3586ebb7ce64ad87e83ee8bfe1cd04ce8dc"}
Sep 12 01:15:03.900043 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:03.900011    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4ef62303112ae4c2c04b3e954aef3586ebb7ce64ad87e83ee8bfe1cd04ce8dc"
Sep 12 01:15:15.029226 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:15.029191    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf_50ff1acc-4e2a-487d-a907-aea16d94f938/payload-processing/0.log"
Sep 12 01:15:15.152617 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:15.152573    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-plxcj_f4fff3f1-7b21-4c2d-a4fc-33d2a733834e/payload-processing/0.log"
Sep 12 01:15:15.270889 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:15.270864    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth_8c239a42-fbfa-42d7-a670-3bf0c77a8c80/payload-processing/0.log"
Sep 12 01:15:23.291036 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.291008    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"]
Sep 12 01:15:23.291424 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.291358    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6ae4a30-e750-45d0-8b72-93ef713187b7" containerName="cleanup"
Sep 12 01:15:23.291424 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.291369    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6ae4a30-e750-45d0-8b72-93ef713187b7" containerName="cleanup"
Sep 12 01:15:23.291424 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.291391    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d06bf9a2-8993-4311-9d28-635593b6ae84" containerName="cleanup"
Sep 12 01:15:23.291424 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.291396    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d06bf9a2-8993-4311-9d28-635593b6ae84" containerName="cleanup"
Sep 12 01:15:23.291646 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.291448    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6ae4a30-e750-45d0-8b72-93ef713187b7" containerName="cleanup"
Sep 12 01:15:23.291646 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.291457    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d06bf9a2-8993-4311-9d28-635593b6ae84" containerName="cleanup"
Sep 12 01:15:23.294661 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.294642    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"
Sep 12 01:15:23.297359 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.297341    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-bfe85d\""
Sep 12 01:15:23.306751 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.306730    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"]
Sep 12 01:15:23.319704 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.319678    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"]
Sep 12 01:15:23.322952 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.322938    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"
Sep 12 01:15:23.325611 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.325580    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-bfe85d-dockercfg-4z69w\""
Sep 12 01:15:23.330856 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.330784    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"]
Sep 12 01:15:23.337059 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.337040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b76b5eb5-019e-4677-b874-75cfd2536f47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm\" (UID: \"b76b5eb5-019e-4677-b874-75cfd2536f47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"
Sep 12 01:15:23.337159 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.337110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4gvm\" (UniqueName: \"kubernetes.io/projected/b76b5eb5-019e-4677-b874-75cfd2536f47-kube-api-access-z4gvm\") pod \"payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm\" (UID: \"b76b5eb5-019e-4677-b874-75cfd2536f47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"
Sep 12 01:15:23.438659 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.438618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e40705a5-5bf9-4cdf-b645-251ff611fdeb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59\" (UID: \"e40705a5-5bf9-4cdf-b645-251ff611fdeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"
Sep 12 01:15:23.438869 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.438702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4gvm\" (UniqueName: \"kubernetes.io/projected/b76b5eb5-019e-4677-b874-75cfd2536f47-kube-api-access-z4gvm\") pod \"payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm\" (UID: \"b76b5eb5-019e-4677-b874-75cfd2536f47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"
Sep 12 01:15:23.438869 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.438759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfwgd\" (UniqueName: \"kubernetes.io/projected/e40705a5-5bf9-4cdf-b645-251ff611fdeb-kube-api-access-pfwgd\") pod \"payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59\" (UID: \"e40705a5-5bf9-4cdf-b645-251ff611fdeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"
Sep 12 01:15:23.438869 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.438800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b76b5eb5-019e-4677-b874-75cfd2536f47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm\" (UID: \"b76b5eb5-019e-4677-b874-75cfd2536f47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"
Sep 12 01:15:23.439768 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.439739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b76b5eb5-019e-4677-b874-75cfd2536f47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm\" (UID: \"b76b5eb5-019e-4677-b874-75cfd2536f47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"
Sep 12 01:15:23.451204 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.451177    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4gvm\" (UniqueName: \"kubernetes.io/projected/b76b5eb5-019e-4677-b874-75cfd2536f47-kube-api-access-z4gvm\") pod \"payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm\" (UID: \"b76b5eb5-019e-4677-b874-75cfd2536f47\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"
Sep 12 01:15:23.539924 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.539882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwgd\" (UniqueName: \"kubernetes.io/projected/e40705a5-5bf9-4cdf-b645-251ff611fdeb-kube-api-access-pfwgd\") pod \"payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59\" (UID: \"e40705a5-5bf9-4cdf-b645-251ff611fdeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"
Sep 12 01:15:23.540078 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.539970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e40705a5-5bf9-4cdf-b645-251ff611fdeb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59\" (UID: \"e40705a5-5bf9-4cdf-b645-251ff611fdeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"
Sep 12 01:15:23.540579 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.540561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e40705a5-5bf9-4cdf-b645-251ff611fdeb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59\" (UID: \"e40705a5-5bf9-4cdf-b645-251ff611fdeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"
Sep 12 01:15:23.550597 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.550543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfwgd\" (UniqueName: \"kubernetes.io/projected/e40705a5-5bf9-4cdf-b645-251ff611fdeb-kube-api-access-pfwgd\") pod \"payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59\" (UID: \"e40705a5-5bf9-4cdf-b645-251ff611fdeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"
Sep 12 01:15:23.605238 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.605209    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"
Sep 12 01:15:23.636023 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.635990    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"
Sep 12 01:15:23.880253 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.880224    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"]
Sep 12 01:15:23.882771 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:15:23.882739    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76b5eb5_019e_4677_b874_75cfd2536f47.slice/crio-25c53fb8fc7c9fc92d10397921fc605b1548a724a8c0ce53d948852f719161c2 WatchSource:0}: Error finding container 25c53fb8fc7c9fc92d10397921fc605b1548a724a8c0ce53d948852f719161c2: Status 404 returned error can't find the container with id 25c53fb8fc7c9fc92d10397921fc605b1548a724a8c0ce53d948852f719161c2
Sep 12 01:15:23.915847 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:23.915824    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"]
Sep 12 01:15:23.919745 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:15:23.919707    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40705a5_5bf9_4cdf_b645_251ff611fdeb.slice/crio-f0d7dbb8ba90c446872e264582b5a6f229ea63856f7a74a8a7974c2585587a59 WatchSource:0}: Error finding container f0d7dbb8ba90c446872e264582b5a6f229ea63856f7a74a8a7974c2585587a59: Status 404 returned error can't find the container with id f0d7dbb8ba90c446872e264582b5a6f229ea63856f7a74a8a7974c2585587a59
Sep 12 01:15:24.006810 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:24.006771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59" event={"ID":"e40705a5-5bf9-4cdf-b645-251ff611fdeb","Type":"ContainerStarted","Data":"0c9bed0f48595d0d1d13288eacb20847d24c80222dbf9e4091951fd10b14150a"}
Sep 12 01:15:24.007015 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:24.006839    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59" event={"ID":"e40705a5-5bf9-4cdf-b645-251ff611fdeb","Type":"ContainerStarted","Data":"f0d7dbb8ba90c446872e264582b5a6f229ea63856f7a74a8a7974c2585587a59"}
Sep 12 01:15:24.007015 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:24.006890    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"
Sep 12 01:15:24.008962 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:24.008932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm" event={"ID":"b76b5eb5-019e-4677-b874-75cfd2536f47","Type":"ContainerStarted","Data":"fb97658dd216ef7577a735c576fca990f30ff34ded1461d11b69914c43decd46"}
Sep 12 01:15:24.009078 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:24.008984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm" event={"ID":"b76b5eb5-019e-4677-b874-75cfd2536f47","Type":"ContainerStarted","Data":"25c53fb8fc7c9fc92d10397921fc605b1548a724a8c0ce53d948852f719161c2"}
Sep 12 01:15:24.009078 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:24.009025    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"
Sep 12 01:15:24.026215 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:24.026169    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59" podStartSLOduration=1.026152601 podStartE2EDuration="1.026152601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:15:24.023084639 +0000 UTC m=+1948.942376428" watchObservedRunningTime="2026-09-12 01:15:24.026152601 +0000 UTC m=+1948.945444437"
Sep 12 01:15:24.039401 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:24.039352    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm" podStartSLOduration=1.039326066 podStartE2EDuration="1.039326066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:15:24.038500129 +0000 UTC m=+1948.957791917" watchObservedRunningTime="2026-09-12 01:15:24.039326066 +0000 UTC m=+1948.958617853"
Sep 12 01:15:35.016656 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:35.016625    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"
Sep 12 01:15:35.017102 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:35.016694    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"
Sep 12 01:15:55.893275 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:55.893206    2579 scope.go:117] "RemoveContainer" containerID="9ebc853fac6407c5393d5c489525a7749ce2e00502580edb58254d64a29f65da"
Sep 12 01:15:55.901515 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:55.901488    2579 scope.go:117] "RemoveContainer" containerID="9decff80074f5bd6cff67ad49cc09edac6419e66e8be6680906133937aebe675"
Sep 12 01:15:55.911062 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:55.911035    2579 scope.go:117] "RemoveContainer" containerID="07736b658343eb398327b6bd18cc6e6eadeec996dd7dedaf279eb955916d7ceb"
Sep 12 01:15:55.918700 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:15:55.918679    2579 scope.go:117] "RemoveContainer" containerID="9c4cf38e0fc7293d99814d65aff22d10af1c0ba54e554fefb04eb02691aa8f3c"
Sep 12 01:16:01.073041 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.073002    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"]
Sep 12 01:16:01.074060 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.073298    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59" podUID="e40705a5-5bf9-4cdf-b645-251ff611fdeb" containerName="payload-processing" containerID="cri-o://0c9bed0f48595d0d1d13288eacb20847d24c80222dbf9e4091951fd10b14150a" gracePeriod=30
Sep 12 01:16:01.082770 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.082744    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"]
Sep 12 01:16:01.083016 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.082968    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm" podUID="b76b5eb5-019e-4677-b874-75cfd2536f47" containerName="payload-pre-processing" containerID="cri-o://fb97658dd216ef7577a735c576fca990f30ff34ded1461d11b69914c43decd46" gracePeriod=30
Sep 12 01:16:01.196037 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.196007    2579 generic.go:356] "Generic (PLEG): container finished" podID="e40705a5-5bf9-4cdf-b645-251ff611fdeb" containerID="0c9bed0f48595d0d1d13288eacb20847d24c80222dbf9e4091951fd10b14150a" exitCode=0
Sep 12 01:16:01.196172 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.196046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59" event={"ID":"e40705a5-5bf9-4cdf-b645-251ff611fdeb","Type":"ContainerDied","Data":"0c9bed0f48595d0d1d13288eacb20847d24c80222dbf9e4091951fd10b14150a"}
Sep 12 01:16:01.339156 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.339132    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"
Sep 12 01:16:01.353762 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.353742    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfwgd\" (UniqueName: \"kubernetes.io/projected/e40705a5-5bf9-4cdf-b645-251ff611fdeb-kube-api-access-pfwgd\") pod \"e40705a5-5bf9-4cdf-b645-251ff611fdeb\" (UID: \"e40705a5-5bf9-4cdf-b645-251ff611fdeb\") "
Sep 12 01:16:01.353857 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.353773    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e40705a5-5bf9-4cdf-b645-251ff611fdeb-plugins-config-volume\") pod \"e40705a5-5bf9-4cdf-b645-251ff611fdeb\" (UID: \"e40705a5-5bf9-4cdf-b645-251ff611fdeb\") "
Sep 12 01:16:01.354116 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.354095    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40705a5-5bf9-4cdf-b645-251ff611fdeb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e40705a5-5bf9-4cdf-b645-251ff611fdeb" (UID: "e40705a5-5bf9-4cdf-b645-251ff611fdeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:16:01.355903 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.355835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40705a5-5bf9-4cdf-b645-251ff611fdeb-kube-api-access-pfwgd" (OuterVolumeSpecName: "kube-api-access-pfwgd") pod "e40705a5-5bf9-4cdf-b645-251ff611fdeb" (UID: "e40705a5-5bf9-4cdf-b645-251ff611fdeb"). InnerVolumeSpecName "kube-api-access-pfwgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:16:01.366227 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.366204    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"
Sep 12 01:16:01.454397 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.454365    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b76b5eb5-019e-4677-b874-75cfd2536f47-plugins-config-volume\") pod \"b76b5eb5-019e-4677-b874-75cfd2536f47\" (UID: \"b76b5eb5-019e-4677-b874-75cfd2536f47\") "
Sep 12 01:16:01.454581 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.454430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4gvm\" (UniqueName: \"kubernetes.io/projected/b76b5eb5-019e-4677-b874-75cfd2536f47-kube-api-access-z4gvm\") pod \"b76b5eb5-019e-4677-b874-75cfd2536f47\" (UID: \"b76b5eb5-019e-4677-b874-75cfd2536f47\") "
Sep 12 01:16:01.454697 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.454586    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e40705a5-5bf9-4cdf-b645-251ff611fdeb-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:01.454697 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.454596    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfwgd\" (UniqueName: \"kubernetes.io/projected/e40705a5-5bf9-4cdf-b645-251ff611fdeb-kube-api-access-pfwgd\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:01.454815 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.454715    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76b5eb5-019e-4677-b874-75cfd2536f47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b76b5eb5-019e-4677-b874-75cfd2536f47" (UID: "b76b5eb5-019e-4677-b874-75cfd2536f47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:16:01.456450 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.456429    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76b5eb5-019e-4677-b874-75cfd2536f47-kube-api-access-z4gvm" (OuterVolumeSpecName: "kube-api-access-z4gvm") pod "b76b5eb5-019e-4677-b874-75cfd2536f47" (UID: "b76b5eb5-019e-4677-b874-75cfd2536f47"). InnerVolumeSpecName "kube-api-access-z4gvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:16:01.555198 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.555167    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b76b5eb5-019e-4677-b874-75cfd2536f47-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:01.555198 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:01.555194    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4gvm\" (UniqueName: \"kubernetes.io/projected/b76b5eb5-019e-4677-b874-75cfd2536f47-kube-api-access-z4gvm\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:02.201947 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:02.201921    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"
Sep 12 01:16:02.202385 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:02.201915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59" event={"ID":"e40705a5-5bf9-4cdf-b645-251ff611fdeb","Type":"ContainerDied","Data":"f0d7dbb8ba90c446872e264582b5a6f229ea63856f7a74a8a7974c2585587a59"}
Sep 12 01:16:02.202385 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:02.202056    2579 scope.go:117] "RemoveContainer" containerID="0c9bed0f48595d0d1d13288eacb20847d24c80222dbf9e4091951fd10b14150a"
Sep 12 01:16:02.203452 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:02.203371    2579 generic.go:356] "Generic (PLEG): container finished" podID="b76b5eb5-019e-4677-b874-75cfd2536f47" containerID="fb97658dd216ef7577a735c576fca990f30ff34ded1461d11b69914c43decd46" exitCode=0
Sep 12 01:16:02.203452 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:02.203418    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"
Sep 12 01:16:02.203582 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:02.203458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm" event={"ID":"b76b5eb5-019e-4677-b874-75cfd2536f47","Type":"ContainerDied","Data":"fb97658dd216ef7577a735c576fca990f30ff34ded1461d11b69914c43decd46"}
Sep 12 01:16:02.203582 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:02.203510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm" event={"ID":"b76b5eb5-019e-4677-b874-75cfd2536f47","Type":"ContainerDied","Data":"25c53fb8fc7c9fc92d10397921fc605b1548a724a8c0ce53d948852f719161c2"}
Sep 12 01:16:02.210958 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:02.210925    2579 scope.go:117] "RemoveContainer" containerID="fb97658dd216ef7577a735c576fca990f30ff34ded1461d11b69914c43decd46"
Sep 12 01:16:02.218339 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:02.218320    2579 scope.go:117] "RemoveContainer" containerID="fb97658dd216ef7577a735c576fca990f30ff34ded1461d11b69914c43decd46"
Sep 12 01:16:02.218681 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:16:02.218660    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb97658dd216ef7577a735c576fca990f30ff34ded1461d11b69914c43decd46\": container with ID starting with fb97658dd216ef7577a735c576fca990f30ff34ded1461d11b69914c43decd46 not found: ID does not exist" containerID="fb97658dd216ef7577a735c576fca990f30ff34ded1461d11b69914c43decd46"
Sep 12 01:16:02.218770 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:02.218692    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb97658dd216ef7577a735c576fca990f30ff34ded1461d11b69914c43decd46"} err="failed to get container status \"fb97658dd216ef7577a735c576fca990f30ff34ded1461d11b69914c43decd46\": rpc error: code = NotFound desc = could not find container \"fb97658dd216ef7577a735c576fca990f30ff34ded1461d11b69914c43decd46\": container with ID starting with fb97658dd216ef7577a735c576fca990f30ff34ded1461d11b69914c43decd46 not found: ID does not exist"
Sep 12 01:16:02.228744 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:02.228719    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"]
Sep 12 01:16:02.230668 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:02.230648    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-bfe85d-d875b66c8-6hksm"]
Sep 12 01:16:02.243850 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:02.243831    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"]
Sep 12 01:16:02.246745 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:02.246725    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-bfe85d-56c597df75-g7t59"]
Sep 12 01:16:03.591262 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:03.591231    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b76b5eb5-019e-4677-b874-75cfd2536f47" path="/var/lib/kubelet/pods/b76b5eb5-019e-4677-b874-75cfd2536f47/volumes"
Sep 12 01:16:03.591674 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:03.591561    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e40705a5-5bf9-4cdf-b645-251ff611fdeb" path="/var/lib/kubelet/pods/e40705a5-5bf9-4cdf-b645-251ff611fdeb/volumes"
Sep 12 01:16:11.680201 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:11.680165    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv"]
Sep 12 01:16:11.680668 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:11.680518    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76b5eb5-019e-4677-b874-75cfd2536f47" containerName="payload-pre-processing"
Sep 12 01:16:11.680668 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:11.680529    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76b5eb5-019e-4677-b874-75cfd2536f47" containerName="payload-pre-processing"
Sep 12 01:16:11.680668 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:11.680549    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e40705a5-5bf9-4cdf-b645-251ff611fdeb" containerName="payload-processing"
Sep 12 01:16:11.680668 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:11.680554    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e40705a5-5bf9-4cdf-b645-251ff611fdeb" containerName="payload-processing"
Sep 12 01:16:11.680668 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:11.680629    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76b5eb5-019e-4677-b874-75cfd2536f47" containerName="payload-pre-processing"
Sep 12 01:16:11.680668 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:11.680641    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e40705a5-5bf9-4cdf-b645-251ff611fdeb" containerName="payload-processing"
Sep 12 01:16:11.685084 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:11.685065    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv"
Sep 12 01:16:11.688159 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:11.688130    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:16:11.688727 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:11.688703    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2rn9k\""
Sep 12 01:16:11.690381 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:11.690358    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv"]
Sep 12 01:16:11.741719 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:11.741687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89c5c2e-7871-46d0-9068-0de66ccd51b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv\" (UID: \"f89c5c2e-7871-46d0-9068-0de66ccd51b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv"
Sep 12 01:16:11.842141 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:11.842105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89c5c2e-7871-46d0-9068-0de66ccd51b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv\" (UID: \"f89c5c2e-7871-46d0-9068-0de66ccd51b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv"
Sep 12 01:16:11.842701 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:11.842683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89c5c2e-7871-46d0-9068-0de66ccd51b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv\" (UID: \"f89c5c2e-7871-46d0-9068-0de66ccd51b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv"
Sep 12 01:16:11.996632 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:11.996533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv"
Sep 12 01:16:12.269115 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:12.269093    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv"]
Sep 12 01:16:12.271458 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:16:12.271431    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89c5c2e_7871_46d0_9068_0de66ccd51b6.slice/crio-35bef299ee40147cf7741ca0ec944530e8f3362caa4770785afec4980d85d37f WatchSource:0}: Error finding container 35bef299ee40147cf7741ca0ec944530e8f3362caa4770785afec4980d85d37f: Status 404 returned error can't find the container with id 35bef299ee40147cf7741ca0ec944530e8f3362caa4770785afec4980d85d37f
Sep 12 01:16:13.260502 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:13.260469    2579 generic.go:356] "Generic (PLEG): container finished" podID="f89c5c2e-7871-46d0-9068-0de66ccd51b6" containerID="9cb7486ae2e1127b5e2be83a8923bc3604f61851e59cf358b5b34a1158fda424" exitCode=0
Sep 12 01:16:13.260852 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:13.260517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv" event={"ID":"f89c5c2e-7871-46d0-9068-0de66ccd51b6","Type":"ContainerDied","Data":"9cb7486ae2e1127b5e2be83a8923bc3604f61851e59cf358b5b34a1158fda424"}
Sep 12 01:16:13.260852 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:13.260561    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv" event={"ID":"f89c5c2e-7871-46d0-9068-0de66ccd51b6","Type":"ContainerStarted","Data":"35bef299ee40147cf7741ca0ec944530e8f3362caa4770785afec4980d85d37f"}
Sep 12 01:16:14.404170 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:14.404140    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv"
Sep 12 01:16:14.466427 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:14.466399    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89c5c2e-7871-46d0-9068-0de66ccd51b6-maas-api-service-ca\") pod \"f89c5c2e-7871-46d0-9068-0de66ccd51b6\" (UID: \"f89c5c2e-7871-46d0-9068-0de66ccd51b6\") "
Sep 12 01:16:14.466789 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:14.466766    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89c5c2e-7871-46d0-9068-0de66ccd51b6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f89c5c2e-7871-46d0-9068-0de66ccd51b6" (UID: "f89c5c2e-7871-46d0-9068-0de66ccd51b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:16:14.567959 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:14.567895    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89c5c2e-7871-46d0-9068-0de66ccd51b6-maas-api-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:15.272179 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:15.272150    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv"
Sep 12 01:16:15.272374 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:15.272145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv" event={"ID":"f89c5c2e-7871-46d0-9068-0de66ccd51b6","Type":"ContainerDied","Data":"35bef299ee40147cf7741ca0ec944530e8f3362caa4770785afec4980d85d37f"}
Sep 12 01:16:15.272374 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:15.272285    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35bef299ee40147cf7741ca0ec944530e8f3362caa4770785afec4980d85d37f"
Sep 12 01:16:27.210539 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.210506    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"]
Sep 12 01:16:27.211027 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.210771    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf" podUID="50ff1acc-4e2a-487d-a907-aea16d94f938" containerName="payload-processing" containerID="cri-o://8a4ec2bb70427942d0a13d71981121847ae2e5dcccf5a7922d77f28fd3d6532b" gracePeriod=30
Sep 12 01:16:27.217656 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.217627    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"]
Sep 12 01:16:27.217835 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.217815    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg" podUID="bd89d965-fd17-4da6-b568-68b2fd2432e0" containerName="payload-pre-processing" containerID="cri-o://7815951057de79acf88a711e8a71f4161ad53599a349a9c64906d0324e600b9f" gracePeriod=30
Sep 12 01:16:27.334977 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.334946    2579 generic.go:356] "Generic (PLEG): container finished" podID="50ff1acc-4e2a-487d-a907-aea16d94f938" containerID="8a4ec2bb70427942d0a13d71981121847ae2e5dcccf5a7922d77f28fd3d6532b" exitCode=0
Sep 12 01:16:27.335109 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.335019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf" event={"ID":"50ff1acc-4e2a-487d-a907-aea16d94f938","Type":"ContainerDied","Data":"8a4ec2bb70427942d0a13d71981121847ae2e5dcccf5a7922d77f28fd3d6532b"}
Sep 12 01:16:27.390625 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.390547    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv"]
Sep 12 01:16:27.393277 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.393250    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5fbbcd-6ltlv"]
Sep 12 01:16:27.467707 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.467642    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"
Sep 12 01:16:27.495990 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.495963    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"
Sep 12 01:16:27.571775 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.571742    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5b2r\" (UniqueName: \"kubernetes.io/projected/50ff1acc-4e2a-487d-a907-aea16d94f938-kube-api-access-j5b2r\") pod \"50ff1acc-4e2a-487d-a907-aea16d94f938\" (UID: \"50ff1acc-4e2a-487d-a907-aea16d94f938\") "
Sep 12 01:16:27.571957 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.571808    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ff1acc-4e2a-487d-a907-aea16d94f938-plugins-config-volume\") pod \"50ff1acc-4e2a-487d-a907-aea16d94f938\" (UID: \"50ff1acc-4e2a-487d-a907-aea16d94f938\") "
Sep 12 01:16:27.571957 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.571840    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd89d965-fd17-4da6-b568-68b2fd2432e0-plugins-config-volume\") pod \"bd89d965-fd17-4da6-b568-68b2fd2432e0\" (UID: \"bd89d965-fd17-4da6-b568-68b2fd2432e0\") "
Sep 12 01:16:27.571957 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.571867    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5vsz\" (UniqueName: \"kubernetes.io/projected/bd89d965-fd17-4da6-b568-68b2fd2432e0-kube-api-access-p5vsz\") pod \"bd89d965-fd17-4da6-b568-68b2fd2432e0\" (UID: \"bd89d965-fd17-4da6-b568-68b2fd2432e0\") "
Sep 12 01:16:27.572283 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.572256    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ff1acc-4e2a-487d-a907-aea16d94f938-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "50ff1acc-4e2a-487d-a907-aea16d94f938" (UID: "50ff1acc-4e2a-487d-a907-aea16d94f938"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:16:27.572359 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.572256    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd89d965-fd17-4da6-b568-68b2fd2432e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bd89d965-fd17-4da6-b568-68b2fd2432e0" (UID: "bd89d965-fd17-4da6-b568-68b2fd2432e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:16:27.573950 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.573924    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd89d965-fd17-4da6-b568-68b2fd2432e0-kube-api-access-p5vsz" (OuterVolumeSpecName: "kube-api-access-p5vsz") pod "bd89d965-fd17-4da6-b568-68b2fd2432e0" (UID: "bd89d965-fd17-4da6-b568-68b2fd2432e0"). InnerVolumeSpecName "kube-api-access-p5vsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:16:27.573950 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.573939    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ff1acc-4e2a-487d-a907-aea16d94f938-kube-api-access-j5b2r" (OuterVolumeSpecName: "kube-api-access-j5b2r") pod "50ff1acc-4e2a-487d-a907-aea16d94f938" (UID: "50ff1acc-4e2a-487d-a907-aea16d94f938"). InnerVolumeSpecName "kube-api-access-j5b2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:16:27.590893 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.590868    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f89c5c2e-7871-46d0-9068-0de66ccd51b6" path="/var/lib/kubelet/pods/f89c5c2e-7871-46d0-9068-0de66ccd51b6/volumes"
Sep 12 01:16:27.672742 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.672713    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ff1acc-4e2a-487d-a907-aea16d94f938-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:27.672742 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.672740    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd89d965-fd17-4da6-b568-68b2fd2432e0-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:27.672971 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.672750    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5vsz\" (UniqueName: \"kubernetes.io/projected/bd89d965-fd17-4da6-b568-68b2fd2432e0-kube-api-access-p5vsz\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:27.672971 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:27.672759    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5b2r\" (UniqueName: \"kubernetes.io/projected/50ff1acc-4e2a-487d-a907-aea16d94f938-kube-api-access-j5b2r\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:16:28.341136 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:28.341096    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"
Sep 12 01:16:28.341588 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:28.341130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf" event={"ID":"50ff1acc-4e2a-487d-a907-aea16d94f938","Type":"ContainerDied","Data":"04fdcba54e2aff34c9da14cc0e5ca3f58852fdfc5a503b06e65af781c9ce65f2"}
Sep 12 01:16:28.341588 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:28.341187    2579 scope.go:117] "RemoveContainer" containerID="8a4ec2bb70427942d0a13d71981121847ae2e5dcccf5a7922d77f28fd3d6532b"
Sep 12 01:16:28.342648 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:28.342623    2579 generic.go:356] "Generic (PLEG): container finished" podID="bd89d965-fd17-4da6-b568-68b2fd2432e0" containerID="7815951057de79acf88a711e8a71f4161ad53599a349a9c64906d0324e600b9f" exitCode=0
Sep 12 01:16:28.342765 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:28.342674    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"
Sep 12 01:16:28.342765 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:28.342691    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg" event={"ID":"bd89d965-fd17-4da6-b568-68b2fd2432e0","Type":"ContainerDied","Data":"7815951057de79acf88a711e8a71f4161ad53599a349a9c64906d0324e600b9f"}
Sep 12 01:16:28.342765 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:28.342735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg" event={"ID":"bd89d965-fd17-4da6-b568-68b2fd2432e0","Type":"ContainerDied","Data":"29551895be8ceaaf108eaf0f67b55a634e71601d26d611e5fe968a19e0eeeaec"}
Sep 12 01:16:28.350045 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:28.350024    2579 scope.go:117] "RemoveContainer" containerID="7815951057de79acf88a711e8a71f4161ad53599a349a9c64906d0324e600b9f"
Sep 12 01:16:28.357450 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:28.357434    2579 scope.go:117] "RemoveContainer" containerID="7815951057de79acf88a711e8a71f4161ad53599a349a9c64906d0324e600b9f"
Sep 12 01:16:28.357753 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:16:28.357734    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7815951057de79acf88a711e8a71f4161ad53599a349a9c64906d0324e600b9f\": container with ID starting with 7815951057de79acf88a711e8a71f4161ad53599a349a9c64906d0324e600b9f not found: ID does not exist" containerID="7815951057de79acf88a711e8a71f4161ad53599a349a9c64906d0324e600b9f"
Sep 12 01:16:28.357810 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:28.357761    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7815951057de79acf88a711e8a71f4161ad53599a349a9c64906d0324e600b9f"} err="failed to get container status \"7815951057de79acf88a711e8a71f4161ad53599a349a9c64906d0324e600b9f\": rpc error: code = NotFound desc = could not find container \"7815951057de79acf88a711e8a71f4161ad53599a349a9c64906d0324e600b9f\": container with ID starting with 7815951057de79acf88a711e8a71f4161ad53599a349a9c64906d0324e600b9f not found: ID does not exist"
Sep 12 01:16:28.364090 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:28.364072    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"]
Sep 12 01:16:28.366982 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:28.366960    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5fbbcd-6f57f548d5-24wdf"]
Sep 12 01:16:28.375722 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:28.375699    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"]
Sep 12 01:16:28.379762 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:28.379737    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5fbbcd-7b54c6f898-6fjqg"]
Sep 12 01:16:29.591468 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:29.591433    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50ff1acc-4e2a-487d-a907-aea16d94f938" path="/var/lib/kubelet/pods/50ff1acc-4e2a-487d-a907-aea16d94f938/volumes"
Sep 12 01:16:29.591831 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:16:29.591816    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd89d965-fd17-4da6-b568-68b2fd2432e0" path="/var/lib/kubelet/pods/bd89d965-fd17-4da6-b568-68b2fd2432e0/volumes"
Sep 12 01:17:16.015412 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.015380    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"]
Sep 12 01:17:16.015903 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.015566    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl" podUID="17426b08-e769-4573-897e-14736c23767f" containerName="istio-proxy" containerID="cri-o://00e115b9870a179d5e3f1b1a40d8e0cf1e1ed0251b0fd2598c57c1aba193d87b" gracePeriod=30
Sep 12 01:17:16.401414 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.401337    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj"]
Sep 12 01:17:16.401722 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.401709    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f89c5c2e-7871-46d0-9068-0de66ccd51b6" containerName="revoke-keys"
Sep 12 01:17:16.401773 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.401724    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f89c5c2e-7871-46d0-9068-0de66ccd51b6" containerName="revoke-keys"
Sep 12 01:17:16.401773 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.401748    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd89d965-fd17-4da6-b568-68b2fd2432e0" containerName="payload-pre-processing"
Sep 12 01:17:16.401773 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.401757    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd89d965-fd17-4da6-b568-68b2fd2432e0" containerName="payload-pre-processing"
Sep 12 01:17:16.401773 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.401767    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ff1acc-4e2a-487d-a907-aea16d94f938" containerName="payload-processing"
Sep 12 01:17:16.401773 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.401772    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ff1acc-4e2a-487d-a907-aea16d94f938" containerName="payload-processing"
Sep 12 01:17:16.401934 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.401825    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="50ff1acc-4e2a-487d-a907-aea16d94f938" containerName="payload-processing"
Sep 12 01:17:16.401934 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.401838    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd89d965-fd17-4da6-b568-68b2fd2432e0" containerName="payload-pre-processing"
Sep 12 01:17:16.401934 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.401846    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f89c5c2e-7871-46d0-9068-0de66ccd51b6" containerName="revoke-keys"
Sep 12 01:17:16.404768 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.404751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj"
Sep 12 01:17:16.407470 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.407442    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:17:16.407470 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.407456    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2rn9k\""
Sep 12 01:17:16.414364 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.414340    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj"]
Sep 12 01:17:16.476025 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.475996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d4a3b13-eeca-470b-b390-7e053d44d8b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj\" (UID: \"1d4a3b13-eeca-470b-b390-7e053d44d8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj"
Sep 12 01:17:16.498463 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.498436    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 12 01:17:16.498628 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.498487    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl" podUID="17426b08-e769-4573-897e-14736c23767f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 01:17:16.580868 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.578377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d4a3b13-eeca-470b-b390-7e053d44d8b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj\" (UID: \"1d4a3b13-eeca-470b-b390-7e053d44d8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj"
Sep 12 01:17:16.580868 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.579220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d4a3b13-eeca-470b-b390-7e053d44d8b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj\" (UID: \"1d4a3b13-eeca-470b-b390-7e053d44d8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj"
Sep 12 01:17:16.716951 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.716917    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj"
Sep 12 01:17:16.991479 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:16.991435    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj"]
Sep 12 01:17:16.993783 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:17:16.993747    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4a3b13_eeca_470b_b390_7e053d44d8b9.slice/crio-a6f92bdbffeeaddad35d430fb3bc60f4eae73cc26c27bbacf1602bff9bf34bc3 WatchSource:0}: Error finding container a6f92bdbffeeaddad35d430fb3bc60f4eae73cc26c27bbacf1602bff9bf34bc3: Status 404 returned error can't find the container with id a6f92bdbffeeaddad35d430fb3bc60f4eae73cc26c27bbacf1602bff9bf34bc3
Sep 12 01:17:17.591171 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:17.591090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj" event={"ID":"1d4a3b13-eeca-470b-b390-7e053d44d8b9","Type":"ContainerStarted","Data":"94a7dfd34efc8a5ed740234da86046c48398056eb2ea8108ef4f25173c4d204b"}
Sep 12 01:17:17.591171 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:17.591138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj" event={"ID":"1d4a3b13-eeca-470b-b390-7e053d44d8b9","Type":"ContainerStarted","Data":"a6f92bdbffeeaddad35d430fb3bc60f4eae73cc26c27bbacf1602bff9bf34bc3"}
Sep 12 01:17:17.602565 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:17.602510    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj" podStartSLOduration=1.6024893329999998 podStartE2EDuration="1.602489333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:17:17.600966337 +0000 UTC m=+2062.520258138" watchObservedRunningTime="2026-09-12 01:17:17.602489333 +0000 UTC m=+2062.521781122"
Sep 12 01:17:18.591244 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:18.591206    2579 generic.go:356] "Generic (PLEG): container finished" podID="1d4a3b13-eeca-470b-b390-7e053d44d8b9" containerID="94a7dfd34efc8a5ed740234da86046c48398056eb2ea8108ef4f25173c4d204b" exitCode=0
Sep 12 01:17:18.591628 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:18.591290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj" event={"ID":"1d4a3b13-eeca-470b-b390-7e053d44d8b9","Type":"ContainerDied","Data":"94a7dfd34efc8a5ed740234da86046c48398056eb2ea8108ef4f25173c4d204b"}
Sep 12 01:17:19.731271 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:19.731244    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj"
Sep 12 01:17:19.805170 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:19.805142    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d4a3b13-eeca-470b-b390-7e053d44d8b9-maas-api-service-ca\") pod \"1d4a3b13-eeca-470b-b390-7e053d44d8b9\" (UID: \"1d4a3b13-eeca-470b-b390-7e053d44d8b9\") "
Sep 12 01:17:19.805486 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:19.805466    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d4a3b13-eeca-470b-b390-7e053d44d8b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d4a3b13-eeca-470b-b390-7e053d44d8b9" (UID: "1d4a3b13-eeca-470b-b390-7e053d44d8b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:17:19.906858 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:19.906777    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d4a3b13-eeca-470b-b390-7e053d44d8b9-maas-api-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:20.600747 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:20.600716    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj"
Sep 12 01:17:20.600747 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:20.600727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj" event={"ID":"1d4a3b13-eeca-470b-b390-7e053d44d8b9","Type":"ContainerDied","Data":"a6f92bdbffeeaddad35d430fb3bc60f4eae73cc26c27bbacf1602bff9bf34bc3"}
Sep 12 01:17:20.600965 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:20.600766    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6f92bdbffeeaddad35d430fb3bc60f4eae73cc26c27bbacf1602bff9bf34bc3"
Sep 12 01:17:21.264747 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.264723    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:17:21.316597 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.316532    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17426b08-e769-4573-897e-14736c23767f-istio-token\") pod \"17426b08-e769-4573-897e-14736c23767f\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") "
Sep 12 01:17:21.316597 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.316576    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-credential-socket\") pod \"17426b08-e769-4573-897e-14736c23767f\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") "
Sep 12 01:17:21.316784 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.316628    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17426b08-e769-4573-897e-14736c23767f-istiod-ca-cert\") pod \"17426b08-e769-4573-897e-14736c23767f\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") "
Sep 12 01:17:21.316784 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.316756    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nhbn\" (UniqueName: \"kubernetes.io/projected/17426b08-e769-4573-897e-14736c23767f-kube-api-access-9nhbn\") pod \"17426b08-e769-4573-897e-14736c23767f\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") "
Sep 12 01:17:21.316897 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.316801    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-istio-envoy\") pod \"17426b08-e769-4573-897e-14736c23767f\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") "
Sep 12 01:17:21.316897 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.316827    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-workload-certs\") pod \"17426b08-e769-4573-897e-14736c23767f\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") "
Sep 12 01:17:21.316897 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.316855    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-istio-data\") pod \"17426b08-e769-4573-897e-14736c23767f\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") "
Sep 12 01:17:21.316897 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.316886    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17426b08-e769-4573-897e-14736c23767f-istio-podinfo\") pod \"17426b08-e769-4573-897e-14736c23767f\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") "
Sep 12 01:17:21.317104 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.316916    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17426b08-e769-4573-897e-14736c23767f" (UID: "17426b08-e769-4573-897e-14736c23767f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:17:21.317104 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.316933    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-workload-socket\") pod \"17426b08-e769-4573-897e-14736c23767f\" (UID: \"17426b08-e769-4573-897e-14736c23767f\") "
Sep 12 01:17:21.317104 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.316959    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17426b08-e769-4573-897e-14736c23767f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17426b08-e769-4573-897e-14736c23767f" (UID: "17426b08-e769-4573-897e-14736c23767f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:17:21.317278 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.317138    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-istio-data" (OuterVolumeSpecName: "istio-data") pod "17426b08-e769-4573-897e-14736c23767f" (UID: "17426b08-e769-4573-897e-14736c23767f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:17:21.317278 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.317211    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-credential-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:21.317278 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.317229    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17426b08-e769-4573-897e-14736c23767f-istiod-ca-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:21.317278 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.317272    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-istio-data\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:21.317477 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.317295    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17426b08-e769-4573-897e-14736c23767f" (UID: "17426b08-e769-4573-897e-14736c23767f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:17:21.317477 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.317375    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17426b08-e769-4573-897e-14736c23767f" (UID: "17426b08-e769-4573-897e-14736c23767f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:17:21.318898 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.318867    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17426b08-e769-4573-897e-14736c23767f" (UID: "17426b08-e769-4573-897e-14736c23767f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:17:21.319000 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.318939    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17426b08-e769-4573-897e-14736c23767f-kube-api-access-9nhbn" (OuterVolumeSpecName: "kube-api-access-9nhbn") pod "17426b08-e769-4573-897e-14736c23767f" (UID: "17426b08-e769-4573-897e-14736c23767f"). InnerVolumeSpecName "kube-api-access-9nhbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:17:21.319139 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.319121    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17426b08-e769-4573-897e-14736c23767f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17426b08-e769-4573-897e-14736c23767f" (UID: "17426b08-e769-4573-897e-14736c23767f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 01:17:21.319139 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.319129    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17426b08-e769-4573-897e-14736c23767f-istio-token" (OuterVolumeSpecName: "istio-token") pod "17426b08-e769-4573-897e-14736c23767f" (UID: "17426b08-e769-4573-897e-14736c23767f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:17:21.417795 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.417770    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nhbn\" (UniqueName: \"kubernetes.io/projected/17426b08-e769-4573-897e-14736c23767f-kube-api-access-9nhbn\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:21.417795 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.417791    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-istio-envoy\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:21.417795 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.417800    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-workload-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:21.417978 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.417808    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17426b08-e769-4573-897e-14736c23767f-istio-podinfo\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:21.417978 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.417817    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17426b08-e769-4573-897e-14736c23767f-workload-socket\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:21.417978 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.417825    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17426b08-e769-4573-897e-14736c23767f-istio-token\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:21.606765 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.606741    2579 generic.go:356] "Generic (PLEG): container finished" podID="17426b08-e769-4573-897e-14736c23767f" containerID="00e115b9870a179d5e3f1b1a40d8e0cf1e1ed0251b0fd2598c57c1aba193d87b" exitCode=0
Sep 12 01:17:21.606899 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.606793    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"
Sep 12 01:17:21.606899 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.606822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl" event={"ID":"17426b08-e769-4573-897e-14736c23767f","Type":"ContainerDied","Data":"00e115b9870a179d5e3f1b1a40d8e0cf1e1ed0251b0fd2598c57c1aba193d87b"}
Sep 12 01:17:21.606899 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.606880    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl" event={"ID":"17426b08-e769-4573-897e-14736c23767f","Type":"ContainerDied","Data":"c8e2a60c4c15dbc6435f77b086e01f2f1a733410310a44cf6344c5e45921fa35"}
Sep 12 01:17:21.607069 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.606908    2579 scope.go:117] "RemoveContainer" containerID="00e115b9870a179d5e3f1b1a40d8e0cf1e1ed0251b0fd2598c57c1aba193d87b"
Sep 12 01:17:21.615555 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.615539    2579 scope.go:117] "RemoveContainer" containerID="26a1e0920a415cfd58f3b304211b834f138e4512ed9388eadd816e636e22131f"
Sep 12 01:17:21.623395 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.623193    2579 scope.go:117] "RemoveContainer" containerID="00e115b9870a179d5e3f1b1a40d8e0cf1e1ed0251b0fd2598c57c1aba193d87b"
Sep 12 01:17:21.623561 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:17:21.623536    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e115b9870a179d5e3f1b1a40d8e0cf1e1ed0251b0fd2598c57c1aba193d87b\": container with ID starting with 00e115b9870a179d5e3f1b1a40d8e0cf1e1ed0251b0fd2598c57c1aba193d87b not found: ID does not exist" containerID="00e115b9870a179d5e3f1b1a40d8e0cf1e1ed0251b0fd2598c57c1aba193d87b"
Sep 12 01:17:21.623700 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.623584    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e115b9870a179d5e3f1b1a40d8e0cf1e1ed0251b0fd2598c57c1aba193d87b"} err="failed to get container status \"00e115b9870a179d5e3f1b1a40d8e0cf1e1ed0251b0fd2598c57c1aba193d87b\": rpc error: code = NotFound desc = could not find container \"00e115b9870a179d5e3f1b1a40d8e0cf1e1ed0251b0fd2598c57c1aba193d87b\": container with ID starting with 00e115b9870a179d5e3f1b1a40d8e0cf1e1ed0251b0fd2598c57c1aba193d87b not found: ID does not exist"
Sep 12 01:17:21.623700 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.623626    2579 scope.go:117] "RemoveContainer" containerID="26a1e0920a415cfd58f3b304211b834f138e4512ed9388eadd816e636e22131f"
Sep 12 01:17:21.623932 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:17:21.623914    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a1e0920a415cfd58f3b304211b834f138e4512ed9388eadd816e636e22131f\": container with ID starting with 26a1e0920a415cfd58f3b304211b834f138e4512ed9388eadd816e636e22131f not found: ID does not exist" containerID="26a1e0920a415cfd58f3b304211b834f138e4512ed9388eadd816e636e22131f"
Sep 12 01:17:21.624003 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.623937    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a1e0920a415cfd58f3b304211b834f138e4512ed9388eadd816e636e22131f"} err="failed to get container status \"26a1e0920a415cfd58f3b304211b834f138e4512ed9388eadd816e636e22131f\": rpc error: code = NotFound desc = could not find container \"26a1e0920a415cfd58f3b304211b834f138e4512ed9388eadd816e636e22131f\": container with ID starting with 26a1e0920a415cfd58f3b304211b834f138e4512ed9388eadd816e636e22131f not found: ID does not exist"
Sep 12 01:17:21.624767 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.624750    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"]
Sep 12 01:17:21.629137 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:21.629116    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5fbbcd-openshift-default-75bc786786-9lmpl"]
Sep 12 01:17:23.590613 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:23.590577    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17426b08-e769-4573-897e-14736c23767f" path="/var/lib/kubelet/pods/17426b08-e769-4573-897e-14736c23767f/volumes"
Sep 12 01:17:26.890132 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:26.890082    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-57b051-29819595-tsb6g"]
Sep 12 01:17:26.892942 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:26.892916    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-57b051-29819595-tsb6g"]
Sep 12 01:17:26.916821 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:26.916790    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"]
Sep 12 01:17:26.917355 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:26.917061    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth" podUID="8c239a42-fbfa-42d7-a670-3bf0c77a8c80" containerName="payload-processing" containerID="cri-o://3409e665ee3f2726cbcbd04267373bd8278ffc24afbbee852e78f3635ddf38bd" gracePeriod=30
Sep 12 01:17:26.943461 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:26.943413    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"]
Sep 12 01:17:26.943737 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:26.943688    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp" podUID="90f67e45-b5e8-4ef5-b796-c4a0f26c207a" containerName="payload-pre-processing" containerID="cri-o://9199580877a9439d4622dccf2ae3a3027fb67f45e7a313b867a8e522af6651b0" gracePeriod=30
Sep 12 01:17:27.091500 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.091462    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj"]
Sep 12 01:17:27.094848 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.094819    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-57b051-k9dqj"]
Sep 12 01:17:27.224040 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.224013    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"
Sep 12 01:17:27.226329 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.226310    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"
Sep 12 01:17:27.363705 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.363672    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90f67e45-b5e8-4ef5-b796-c4a0f26c207a-plugins-config-volume\") pod \"90f67e45-b5e8-4ef5-b796-c4a0f26c207a\" (UID: \"90f67e45-b5e8-4ef5-b796-c4a0f26c207a\") "
Sep 12 01:17:27.363869 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.363738    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx9qg\" (UniqueName: \"kubernetes.io/projected/8c239a42-fbfa-42d7-a670-3bf0c77a8c80-kube-api-access-qx9qg\") pod \"8c239a42-fbfa-42d7-a670-3bf0c77a8c80\" (UID: \"8c239a42-fbfa-42d7-a670-3bf0c77a8c80\") "
Sep 12 01:17:27.363869 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.363758    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c239a42-fbfa-42d7-a670-3bf0c77a8c80-plugins-config-volume\") pod \"8c239a42-fbfa-42d7-a670-3bf0c77a8c80\" (UID: \"8c239a42-fbfa-42d7-a670-3bf0c77a8c80\") "
Sep 12 01:17:27.363869 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.363774    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpp8l\" (UniqueName: \"kubernetes.io/projected/90f67e45-b5e8-4ef5-b796-c4a0f26c207a-kube-api-access-gpp8l\") pod \"90f67e45-b5e8-4ef5-b796-c4a0f26c207a\" (UID: \"90f67e45-b5e8-4ef5-b796-c4a0f26c207a\") "
Sep 12 01:17:27.364109 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.364085    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c239a42-fbfa-42d7-a670-3bf0c77a8c80-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8c239a42-fbfa-42d7-a670-3bf0c77a8c80" (UID: "8c239a42-fbfa-42d7-a670-3bf0c77a8c80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:17:27.364165 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.364086    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f67e45-b5e8-4ef5-b796-c4a0f26c207a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90f67e45-b5e8-4ef5-b796-c4a0f26c207a" (UID: "90f67e45-b5e8-4ef5-b796-c4a0f26c207a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:17:27.365782 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.365760    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c239a42-fbfa-42d7-a670-3bf0c77a8c80-kube-api-access-qx9qg" (OuterVolumeSpecName: "kube-api-access-qx9qg") pod "8c239a42-fbfa-42d7-a670-3bf0c77a8c80" (UID: "8c239a42-fbfa-42d7-a670-3bf0c77a8c80"). InnerVolumeSpecName "kube-api-access-qx9qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:17:27.365900 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.365881    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f67e45-b5e8-4ef5-b796-c4a0f26c207a-kube-api-access-gpp8l" (OuterVolumeSpecName: "kube-api-access-gpp8l") pod "90f67e45-b5e8-4ef5-b796-c4a0f26c207a" (UID: "90f67e45-b5e8-4ef5-b796-c4a0f26c207a"). InnerVolumeSpecName "kube-api-access-gpp8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:17:27.464519 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.464484    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx9qg\" (UniqueName: \"kubernetes.io/projected/8c239a42-fbfa-42d7-a670-3bf0c77a8c80-kube-api-access-qx9qg\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:27.464519 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.464516    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c239a42-fbfa-42d7-a670-3bf0c77a8c80-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:27.464749 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.464530    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpp8l\" (UniqueName: \"kubernetes.io/projected/90f67e45-b5e8-4ef5-b796-c4a0f26c207a-kube-api-access-gpp8l\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:27.464749 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.464542    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90f67e45-b5e8-4ef5-b796-c4a0f26c207a-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:27.591306 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.591263    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d4a3b13-eeca-470b-b390-7e053d44d8b9" path="/var/lib/kubelet/pods/1d4a3b13-eeca-470b-b390-7e053d44d8b9/volumes"
Sep 12 01:17:27.591659 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.591629    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d06bf9a2-8993-4311-9d28-635593b6ae84" path="/var/lib/kubelet/pods/d06bf9a2-8993-4311-9d28-635593b6ae84/volumes"
Sep 12 01:17:27.635837 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.635811    2579 generic.go:356] "Generic (PLEG): container finished" podID="90f67e45-b5e8-4ef5-b796-c4a0f26c207a" containerID="9199580877a9439d4622dccf2ae3a3027fb67f45e7a313b867a8e522af6651b0" exitCode=0
Sep 12 01:17:27.635963 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.635858    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"
Sep 12 01:17:27.635963 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.635890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp" event={"ID":"90f67e45-b5e8-4ef5-b796-c4a0f26c207a","Type":"ContainerDied","Data":"9199580877a9439d4622dccf2ae3a3027fb67f45e7a313b867a8e522af6651b0"}
Sep 12 01:17:27.635963 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.635951    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp" event={"ID":"90f67e45-b5e8-4ef5-b796-c4a0f26c207a","Type":"ContainerDied","Data":"c65eb2de30484a3f98b029db7c6bf9d6471e0be70da025a6b0c700d47d83eea9"}
Sep 12 01:17:27.636129 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.635985    2579 scope.go:117] "RemoveContainer" containerID="9199580877a9439d4622dccf2ae3a3027fb67f45e7a313b867a8e522af6651b0"
Sep 12 01:17:27.637327 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.637304    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c239a42-fbfa-42d7-a670-3bf0c77a8c80" containerID="3409e665ee3f2726cbcbd04267373bd8278ffc24afbbee852e78f3635ddf38bd" exitCode=0
Sep 12 01:17:27.637421 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.637350    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"
Sep 12 01:17:27.637421 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.637355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth" event={"ID":"8c239a42-fbfa-42d7-a670-3bf0c77a8c80","Type":"ContainerDied","Data":"3409e665ee3f2726cbcbd04267373bd8278ffc24afbbee852e78f3635ddf38bd"}
Sep 12 01:17:27.637421 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.637417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth" event={"ID":"8c239a42-fbfa-42d7-a670-3bf0c77a8c80","Type":"ContainerDied","Data":"69d2c104b2f3ed98917f9c248c0b4fc4c231127c514ce631f90bf2ba59df9f15"}
Sep 12 01:17:27.645037 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.644995    2579 scope.go:117] "RemoveContainer" containerID="9199580877a9439d4622dccf2ae3a3027fb67f45e7a313b867a8e522af6651b0"
Sep 12 01:17:27.645313 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:17:27.645293    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9199580877a9439d4622dccf2ae3a3027fb67f45e7a313b867a8e522af6651b0\": container with ID starting with 9199580877a9439d4622dccf2ae3a3027fb67f45e7a313b867a8e522af6651b0 not found: ID does not exist" containerID="9199580877a9439d4622dccf2ae3a3027fb67f45e7a313b867a8e522af6651b0"
Sep 12 01:17:27.645390 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.645325    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9199580877a9439d4622dccf2ae3a3027fb67f45e7a313b867a8e522af6651b0"} err="failed to get container status \"9199580877a9439d4622dccf2ae3a3027fb67f45e7a313b867a8e522af6651b0\": rpc error: code = NotFound desc = could not find container \"9199580877a9439d4622dccf2ae3a3027fb67f45e7a313b867a8e522af6651b0\": container with ID starting with 9199580877a9439d4622dccf2ae3a3027fb67f45e7a313b867a8e522af6651b0 not found: ID does not exist"
Sep 12 01:17:27.645390 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.645349    2579 scope.go:117] "RemoveContainer" containerID="3409e665ee3f2726cbcbd04267373bd8278ffc24afbbee852e78f3635ddf38bd"
Sep 12 01:17:27.653027 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.653006    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"]
Sep 12 01:17:27.653871 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.653850    2579 scope.go:117] "RemoveContainer" containerID="3409e665ee3f2726cbcbd04267373bd8278ffc24afbbee852e78f3635ddf38bd"
Sep 12 01:17:27.654172 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:17:27.654147    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3409e665ee3f2726cbcbd04267373bd8278ffc24afbbee852e78f3635ddf38bd\": container with ID starting with 3409e665ee3f2726cbcbd04267373bd8278ffc24afbbee852e78f3635ddf38bd not found: ID does not exist" containerID="3409e665ee3f2726cbcbd04267373bd8278ffc24afbbee852e78f3635ddf38bd"
Sep 12 01:17:27.654312 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.654289    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3409e665ee3f2726cbcbd04267373bd8278ffc24afbbee852e78f3635ddf38bd"} err="failed to get container status \"3409e665ee3f2726cbcbd04267373bd8278ffc24afbbee852e78f3635ddf38bd\": rpc error: code = NotFound desc = could not find container \"3409e665ee3f2726cbcbd04267373bd8278ffc24afbbee852e78f3635ddf38bd\": container with ID starting with 3409e665ee3f2726cbcbd04267373bd8278ffc24afbbee852e78f3635ddf38bd not found: ID does not exist"
Sep 12 01:17:27.656402 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.656381    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-57b051-776d95fb58-8s5zp"]
Sep 12 01:17:27.668369 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.668345    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"]
Sep 12 01:17:27.672014 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:27.671994    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-57b051-5944bdfcc9-dtkth"]
Sep 12 01:17:29.590931 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:29.590900    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c239a42-fbfa-42d7-a670-3bf0c77a8c80" path="/var/lib/kubelet/pods/8c239a42-fbfa-42d7-a670-3bf0c77a8c80/volumes"
Sep 12 01:17:29.591295 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:29.591235    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90f67e45-b5e8-4ef5-b796-c4a0f26c207a" path="/var/lib/kubelet/pods/90f67e45-b5e8-4ef5-b796-c4a0f26c207a/volumes"
Sep 12 01:17:34.003099 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.003066    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr"]
Sep 12 01:17:34.003558 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.003540    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c239a42-fbfa-42d7-a670-3bf0c77a8c80" containerName="payload-processing"
Sep 12 01:17:34.003642 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.003560    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c239a42-fbfa-42d7-a670-3bf0c77a8c80" containerName="payload-processing"
Sep 12 01:17:34.003642 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.003578    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f67e45-b5e8-4ef5-b796-c4a0f26c207a" containerName="payload-pre-processing"
Sep 12 01:17:34.003642 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.003588    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f67e45-b5e8-4ef5-b796-c4a0f26c207a" containerName="payload-pre-processing"
Sep 12 01:17:34.003642 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.003625    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17426b08-e769-4573-897e-14736c23767f" containerName="istio-proxy"
Sep 12 01:17:34.003642 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.003634    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="17426b08-e769-4573-897e-14736c23767f" containerName="istio-proxy"
Sep 12 01:17:34.003844 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.003649    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17426b08-e769-4573-897e-14736c23767f" containerName="istio-proxy"
Sep 12 01:17:34.003844 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.003657    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="17426b08-e769-4573-897e-14736c23767f" containerName="istio-proxy"
Sep 12 01:17:34.003844 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.003668    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d4a3b13-eeca-470b-b390-7e053d44d8b9" containerName="revoke-keys"
Sep 12 01:17:34.003844 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.003676    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d4a3b13-eeca-470b-b390-7e053d44d8b9" containerName="revoke-keys"
Sep 12 01:17:34.003844 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.003786    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="17426b08-e769-4573-897e-14736c23767f" containerName="istio-proxy"
Sep 12 01:17:34.003844 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.003799    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f67e45-b5e8-4ef5-b796-c4a0f26c207a" containerName="payload-pre-processing"
Sep 12 01:17:34.003844 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.003810    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d4a3b13-eeca-470b-b390-7e053d44d8b9" containerName="revoke-keys"
Sep 12 01:17:34.003844 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.003819    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c239a42-fbfa-42d7-a670-3bf0c77a8c80" containerName="payload-processing"
Sep 12 01:17:34.006945 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.006927    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr"
Sep 12 01:17:34.009448 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.009426    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2rn9k\""
Sep 12 01:17:34.009448 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.009430    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:17:34.013126 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.013096    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr"]
Sep 12 01:17:34.119912 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.119875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/464c225d-f0d8-4433-8b76-911c88a8273b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr\" (UID: \"464c225d-f0d8-4433-8b76-911c88a8273b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr"
Sep 12 01:17:34.220786 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.220746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/464c225d-f0d8-4433-8b76-911c88a8273b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr\" (UID: \"464c225d-f0d8-4433-8b76-911c88a8273b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr"
Sep 12 01:17:34.221339 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.221320    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/464c225d-f0d8-4433-8b76-911c88a8273b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr\" (UID: \"464c225d-f0d8-4433-8b76-911c88a8273b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr"
Sep 12 01:17:34.319592 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.319501    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr"
Sep 12 01:17:34.595689 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.595664    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr"]
Sep 12 01:17:34.597646 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:17:34.597621    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464c225d_f0d8_4433_8b76_911c88a8273b.slice/crio-6d9b74602ad4cce933da1a4047841216bbcecbd20ef3a73603f23ac1f4b3dde3 WatchSource:0}: Error finding container 6d9b74602ad4cce933da1a4047841216bbcecbd20ef3a73603f23ac1f4b3dde3: Status 404 returned error can't find the container with id 6d9b74602ad4cce933da1a4047841216bbcecbd20ef3a73603f23ac1f4b3dde3
Sep 12 01:17:34.673581 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:34.673544    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr" event={"ID":"464c225d-f0d8-4433-8b76-911c88a8273b","Type":"ContainerStarted","Data":"6d9b74602ad4cce933da1a4047841216bbcecbd20ef3a73603f23ac1f4b3dde3"}
Sep 12 01:17:35.679913 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:35.679880    2579 generic.go:356] "Generic (PLEG): container finished" podID="464c225d-f0d8-4433-8b76-911c88a8273b" containerID="7762dee6a6e8992037f25c3045821e63ad9b7300800c2e409e1fa2c9699ed416" exitCode=0
Sep 12 01:17:35.680256 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:35.679968    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr" event={"ID":"464c225d-f0d8-4433-8b76-911c88a8273b","Type":"ContainerDied","Data":"7762dee6a6e8992037f25c3045821e63ad9b7300800c2e409e1fa2c9699ed416"}
Sep 12 01:17:36.812147 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:36.812125    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr"
Sep 12 01:17:36.945690 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:36.945595    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/464c225d-f0d8-4433-8b76-911c88a8273b-maas-api-service-ca\") pod \"464c225d-f0d8-4433-8b76-911c88a8273b\" (UID: \"464c225d-f0d8-4433-8b76-911c88a8273b\") "
Sep 12 01:17:36.945933 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:36.945914    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464c225d-f0d8-4433-8b76-911c88a8273b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "464c225d-f0d8-4433-8b76-911c88a8273b" (UID: "464c225d-f0d8-4433-8b76-911c88a8273b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:17:37.047203 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:37.047167    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/464c225d-f0d8-4433-8b76-911c88a8273b-maas-api-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:37.691177 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:37.691147    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr"
Sep 12 01:17:37.691177 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:37.691168    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr" event={"ID":"464c225d-f0d8-4433-8b76-911c88a8273b","Type":"ContainerDied","Data":"6d9b74602ad4cce933da1a4047841216bbcecbd20ef3a73603f23ac1f4b3dde3"}
Sep 12 01:17:37.691386 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:37.691216    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d9b74602ad4cce933da1a4047841216bbcecbd20ef3a73603f23ac1f4b3dde3"
Sep 12 01:17:44.631495 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:44.631456    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"]
Sep 12 01:17:44.631899 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:44.631729    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x" podUID="41bc22c2-5fda-4692-aa5b-ddb8ac02be3c" containerName="payload-processing" containerID="cri-o://2c52604f2c372646b064ae5d700e478756bdf6209f2d115676cac6bc484b3649" gracePeriod=30
Sep 12 01:17:44.636081 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:44.636049    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"]
Sep 12 01:17:44.636336 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:44.636236    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g" podUID="35321de5-f1d0-47f2-b947-577c12191d0d" containerName="payload-pre-processing" containerID="cri-o://ec395f18be853fb063708e079be83801b259ec38c17343799ec35caa6ee948c4" gracePeriod=30
Sep 12 01:17:44.818372 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:44.818339    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr"]
Sep 12 01:17:44.821859 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:44.821834    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-61f73f-2x5gr"]
Sep 12 01:17:44.917063 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:44.917035    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"
Sep 12 01:17:44.918975 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:44.918956    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"
Sep 12 01:17:45.008840 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.008807    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35321de5-f1d0-47f2-b947-577c12191d0d-plugins-config-volume\") pod \"35321de5-f1d0-47f2-b947-577c12191d0d\" (UID: \"35321de5-f1d0-47f2-b947-577c12191d0d\") "
Sep 12 01:17:45.008840 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.008844    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl4zf\" (UniqueName: \"kubernetes.io/projected/41bc22c2-5fda-4692-aa5b-ddb8ac02be3c-kube-api-access-bl4zf\") pod \"41bc22c2-5fda-4692-aa5b-ddb8ac02be3c\" (UID: \"41bc22c2-5fda-4692-aa5b-ddb8ac02be3c\") "
Sep 12 01:17:45.009102 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.008869    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41bc22c2-5fda-4692-aa5b-ddb8ac02be3c-plugins-config-volume\") pod \"41bc22c2-5fda-4692-aa5b-ddb8ac02be3c\" (UID: \"41bc22c2-5fda-4692-aa5b-ddb8ac02be3c\") "
Sep 12 01:17:45.009102 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.008926    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zl6t\" (UniqueName: \"kubernetes.io/projected/35321de5-f1d0-47f2-b947-577c12191d0d-kube-api-access-5zl6t\") pod \"35321de5-f1d0-47f2-b947-577c12191d0d\" (UID: \"35321de5-f1d0-47f2-b947-577c12191d0d\") "
Sep 12 01:17:45.009321 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.009296    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bc22c2-5fda-4692-aa5b-ddb8ac02be3c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41bc22c2-5fda-4692-aa5b-ddb8ac02be3c" (UID: "41bc22c2-5fda-4692-aa5b-ddb8ac02be3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:17:45.009411 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.009386    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35321de5-f1d0-47f2-b947-577c12191d0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "35321de5-f1d0-47f2-b947-577c12191d0d" (UID: "35321de5-f1d0-47f2-b947-577c12191d0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:17:45.011091 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.011069    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35321de5-f1d0-47f2-b947-577c12191d0d-kube-api-access-5zl6t" (OuterVolumeSpecName: "kube-api-access-5zl6t") pod "35321de5-f1d0-47f2-b947-577c12191d0d" (UID: "35321de5-f1d0-47f2-b947-577c12191d0d"). InnerVolumeSpecName "kube-api-access-5zl6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:17:45.011401 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.011370    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41bc22c2-5fda-4692-aa5b-ddb8ac02be3c-kube-api-access-bl4zf" (OuterVolumeSpecName: "kube-api-access-bl4zf") pod "41bc22c2-5fda-4692-aa5b-ddb8ac02be3c" (UID: "41bc22c2-5fda-4692-aa5b-ddb8ac02be3c"). InnerVolumeSpecName "kube-api-access-bl4zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:17:45.110378 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.110336    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zl6t\" (UniqueName: \"kubernetes.io/projected/35321de5-f1d0-47f2-b947-577c12191d0d-kube-api-access-5zl6t\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:45.110378 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.110368    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35321de5-f1d0-47f2-b947-577c12191d0d-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:45.110378 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.110385    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl4zf\" (UniqueName: \"kubernetes.io/projected/41bc22c2-5fda-4692-aa5b-ddb8ac02be3c-kube-api-access-bl4zf\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:45.110702 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.110400    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41bc22c2-5fda-4692-aa5b-ddb8ac02be3c-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:45.590720 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.590678    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="464c225d-f0d8-4433-8b76-911c88a8273b" path="/var/lib/kubelet/pods/464c225d-f0d8-4433-8b76-911c88a8273b/volumes"
Sep 12 01:17:45.728319 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.728285    2579 generic.go:356] "Generic (PLEG): container finished" podID="41bc22c2-5fda-4692-aa5b-ddb8ac02be3c" containerID="2c52604f2c372646b064ae5d700e478756bdf6209f2d115676cac6bc484b3649" exitCode=0
Sep 12 01:17:45.728705 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.728334    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"
Sep 12 01:17:45.728705 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.728372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x" event={"ID":"41bc22c2-5fda-4692-aa5b-ddb8ac02be3c","Type":"ContainerDied","Data":"2c52604f2c372646b064ae5d700e478756bdf6209f2d115676cac6bc484b3649"}
Sep 12 01:17:45.728705 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.728445    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x" event={"ID":"41bc22c2-5fda-4692-aa5b-ddb8ac02be3c","Type":"ContainerDied","Data":"746309dfac627feeedc4437d7795d589638f05e88c7dd542dc2d5d9e59a39fca"}
Sep 12 01:17:45.728705 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.728484    2579 scope.go:117] "RemoveContainer" containerID="2c52604f2c372646b064ae5d700e478756bdf6209f2d115676cac6bc484b3649"
Sep 12 01:17:45.729987 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.729774    2579 generic.go:356] "Generic (PLEG): container finished" podID="35321de5-f1d0-47f2-b947-577c12191d0d" containerID="ec395f18be853fb063708e079be83801b259ec38c17343799ec35caa6ee948c4" exitCode=0
Sep 12 01:17:45.729987 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.729830    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"
Sep 12 01:17:45.729987 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.729886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g" event={"ID":"35321de5-f1d0-47f2-b947-577c12191d0d","Type":"ContainerDied","Data":"ec395f18be853fb063708e079be83801b259ec38c17343799ec35caa6ee948c4"}
Sep 12 01:17:45.729987 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.729946    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g" event={"ID":"35321de5-f1d0-47f2-b947-577c12191d0d","Type":"ContainerDied","Data":"ab0f87bebcf5437e6d44ee203da50d5927e558956dd0bae7bcab1ced60d20cbd"}
Sep 12 01:17:45.738216 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.738199    2579 scope.go:117] "RemoveContainer" containerID="2c52604f2c372646b064ae5d700e478756bdf6209f2d115676cac6bc484b3649"
Sep 12 01:17:45.738468 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:17:45.738447    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c52604f2c372646b064ae5d700e478756bdf6209f2d115676cac6bc484b3649\": container with ID starting with 2c52604f2c372646b064ae5d700e478756bdf6209f2d115676cac6bc484b3649 not found: ID does not exist" containerID="2c52604f2c372646b064ae5d700e478756bdf6209f2d115676cac6bc484b3649"
Sep 12 01:17:45.738558 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.738478    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c52604f2c372646b064ae5d700e478756bdf6209f2d115676cac6bc484b3649"} err="failed to get container status \"2c52604f2c372646b064ae5d700e478756bdf6209f2d115676cac6bc484b3649\": rpc error: code = NotFound desc = could not find container \"2c52604f2c372646b064ae5d700e478756bdf6209f2d115676cac6bc484b3649\": container with ID starting with 2c52604f2c372646b064ae5d700e478756bdf6209f2d115676cac6bc484b3649 not found: ID does not exist"
Sep 12 01:17:45.738558 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.738495    2579 scope.go:117] "RemoveContainer" containerID="ec395f18be853fb063708e079be83801b259ec38c17343799ec35caa6ee948c4"
Sep 12 01:17:45.748300 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.746899    2579 scope.go:117] "RemoveContainer" containerID="ec395f18be853fb063708e079be83801b259ec38c17343799ec35caa6ee948c4"
Sep 12 01:17:45.748531 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:17:45.748464    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec395f18be853fb063708e079be83801b259ec38c17343799ec35caa6ee948c4\": container with ID starting with ec395f18be853fb063708e079be83801b259ec38c17343799ec35caa6ee948c4 not found: ID does not exist" containerID="ec395f18be853fb063708e079be83801b259ec38c17343799ec35caa6ee948c4"
Sep 12 01:17:45.748531 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.748495    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec395f18be853fb063708e079be83801b259ec38c17343799ec35caa6ee948c4"} err="failed to get container status \"ec395f18be853fb063708e079be83801b259ec38c17343799ec35caa6ee948c4\": rpc error: code = NotFound desc = could not find container \"ec395f18be853fb063708e079be83801b259ec38c17343799ec35caa6ee948c4\": container with ID starting with ec395f18be853fb063708e079be83801b259ec38c17343799ec35caa6ee948c4 not found: ID does not exist"
Sep 12 01:17:45.749785 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.749765    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"]
Sep 12 01:17:45.753431 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.753408    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-61f73f-64cb7cc75d-s776g"]
Sep 12 01:17:45.764675 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.764656    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"]
Sep 12 01:17:45.768249 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:45.768229    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-61f73f-86c4868989-2rt7x"]
Sep 12 01:17:47.590723 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:47.590691    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35321de5-f1d0-47f2-b947-577c12191d0d" path="/var/lib/kubelet/pods/35321de5-f1d0-47f2-b947-577c12191d0d/volumes"
Sep 12 01:17:47.591219 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:47.591199    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41bc22c2-5fda-4692-aa5b-ddb8ac02be3c" path="/var/lib/kubelet/pods/41bc22c2-5fda-4692-aa5b-ddb8ac02be3c/volumes"
Sep 12 01:17:52.132569 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.132535    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g"]
Sep 12 01:17:52.133054 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.133013    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464c225d-f0d8-4433-8b76-911c88a8273b" containerName="revoke-keys"
Sep 12 01:17:52.133054 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.133032    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="464c225d-f0d8-4433-8b76-911c88a8273b" containerName="revoke-keys"
Sep 12 01:17:52.133054 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.133043    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bc22c2-5fda-4692-aa5b-ddb8ac02be3c" containerName="payload-processing"
Sep 12 01:17:52.133054 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.133052    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bc22c2-5fda-4692-aa5b-ddb8ac02be3c" containerName="payload-processing"
Sep 12 01:17:52.133273 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.133072    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35321de5-f1d0-47f2-b947-577c12191d0d" containerName="payload-pre-processing"
Sep 12 01:17:52.133273 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.133082    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="35321de5-f1d0-47f2-b947-577c12191d0d" containerName="payload-pre-processing"
Sep 12 01:17:52.133273 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.133153    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="35321de5-f1d0-47f2-b947-577c12191d0d" containerName="payload-pre-processing"
Sep 12 01:17:52.133273 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.133166    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="464c225d-f0d8-4433-8b76-911c88a8273b" containerName="revoke-keys"
Sep 12 01:17:52.133273 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.133178    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41bc22c2-5fda-4692-aa5b-ddb8ac02be3c" containerName="payload-processing"
Sep 12 01:17:52.133273 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.133188    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="17426b08-e769-4573-897e-14736c23767f" containerName="istio-proxy"
Sep 12 01:17:52.137572 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.137551    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g"
Sep 12 01:17:52.141557 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.141541    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:17:52.141810 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.141793    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2rn9k\""
Sep 12 01:17:52.147461 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.147440    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g"]
Sep 12 01:17:52.264908 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.264880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0d5199-463b-4127-97c3-f68c81c9b7f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g\" (UID: \"af0d5199-463b-4127-97c3-f68c81c9b7f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g"
Sep 12 01:17:52.365892 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.365856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0d5199-463b-4127-97c3-f68c81c9b7f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g\" (UID: \"af0d5199-463b-4127-97c3-f68c81c9b7f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g"
Sep 12 01:17:52.366466 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.366448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0d5199-463b-4127-97c3-f68c81c9b7f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g\" (UID: \"af0d5199-463b-4127-97c3-f68c81c9b7f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g"
Sep 12 01:17:52.449069 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.449035    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g"
Sep 12 01:17:52.738975 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.738908    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g"]
Sep 12 01:17:52.741343 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:17:52.741314    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0d5199_463b_4127_97c3_f68c81c9b7f7.slice/crio-e9083984e88af5ab08148bc0f3de94b395d58402dcb4c1888afc5cfd166accbb WatchSource:0}: Error finding container e9083984e88af5ab08148bc0f3de94b395d58402dcb4c1888afc5cfd166accbb: Status 404 returned error can't find the container with id e9083984e88af5ab08148bc0f3de94b395d58402dcb4c1888afc5cfd166accbb
Sep 12 01:17:52.773185 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:52.773146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g" event={"ID":"af0d5199-463b-4127-97c3-f68c81c9b7f7","Type":"ContainerStarted","Data":"e9083984e88af5ab08148bc0f3de94b395d58402dcb4c1888afc5cfd166accbb"}
Sep 12 01:17:53.779192 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:53.779103    2579 generic.go:356] "Generic (PLEG): container finished" podID="af0d5199-463b-4127-97c3-f68c81c9b7f7" containerID="4036b21c0437c77be1f4819160852a41eaef28c8be22daad8c8a5d032aacb5fa" exitCode=0
Sep 12 01:17:53.779192 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:53.779166    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g" event={"ID":"af0d5199-463b-4127-97c3-f68c81c9b7f7","Type":"ContainerDied","Data":"4036b21c0437c77be1f4819160852a41eaef28c8be22daad8c8a5d032aacb5fa"}
Sep 12 01:17:54.907333 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:54.907309    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g"
Sep 12 01:17:55.091142 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:55.091063    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0d5199-463b-4127-97c3-f68c81c9b7f7-maas-api-service-ca\") pod \"af0d5199-463b-4127-97c3-f68c81c9b7f7\" (UID: \"af0d5199-463b-4127-97c3-f68c81c9b7f7\") "
Sep 12 01:17:55.091439 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:55.091419    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0d5199-463b-4127-97c3-f68c81c9b7f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af0d5199-463b-4127-97c3-f68c81c9b7f7" (UID: "af0d5199-463b-4127-97c3-f68c81c9b7f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:17:55.192403 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:55.192360    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0d5199-463b-4127-97c3-f68c81c9b7f7-maas-api-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:17:55.670388 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:55.670361    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 01:17:55.673186 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:55.673166    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 01:17:55.788518 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:55.788489    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g"
Sep 12 01:17:55.788685 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:55.788512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g" event={"ID":"af0d5199-463b-4127-97c3-f68c81c9b7f7","Type":"ContainerDied","Data":"e9083984e88af5ab08148bc0f3de94b395d58402dcb4c1888afc5cfd166accbb"}
Sep 12 01:17:55.788685 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:17:55.788568    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9083984e88af5ab08148bc0f3de94b395d58402dcb4c1888afc5cfd166accbb"
Sep 12 01:18:02.674481 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:02.674439    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"]
Sep 12 01:18:02.674978 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:02.674709    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc" podUID="286ac2d9-4dff-4b4c-8db2-78e0a944c990" containerName="payload-pre-processing" containerID="cri-o://2b39f99367d1254bdbb3ab8a64a2af008ab298990137549380ea5b60e9df5589" gracePeriod=30
Sep 12 01:18:02.821139 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:02.821108    2579 generic.go:356] "Generic (PLEG): container finished" podID="286ac2d9-4dff-4b4c-8db2-78e0a944c990" containerID="2b39f99367d1254bdbb3ab8a64a2af008ab298990137549380ea5b60e9df5589" exitCode=0
Sep 12 01:18:02.821294 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:02.821145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc" event={"ID":"286ac2d9-4dff-4b4c-8db2-78e0a944c990","Type":"ContainerDied","Data":"2b39f99367d1254bdbb3ab8a64a2af008ab298990137549380ea5b60e9df5589"}
Sep 12 01:18:02.843728 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:02.843519    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g"]
Sep 12 01:18:02.846446 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:02.846420    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ef5997-gkq9g"]
Sep 12 01:18:02.922251 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:02.922228    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"
Sep 12 01:18:03.051911 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:03.051823    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/286ac2d9-4dff-4b4c-8db2-78e0a944c990-plugins-config-volume\") pod \"286ac2d9-4dff-4b4c-8db2-78e0a944c990\" (UID: \"286ac2d9-4dff-4b4c-8db2-78e0a944c990\") "
Sep 12 01:18:03.052058 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:03.051931    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j98hn\" (UniqueName: \"kubernetes.io/projected/286ac2d9-4dff-4b4c-8db2-78e0a944c990-kube-api-access-j98hn\") pod \"286ac2d9-4dff-4b4c-8db2-78e0a944c990\" (UID: \"286ac2d9-4dff-4b4c-8db2-78e0a944c990\") "
Sep 12 01:18:03.052182 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:03.052158    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/286ac2d9-4dff-4b4c-8db2-78e0a944c990-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "286ac2d9-4dff-4b4c-8db2-78e0a944c990" (UID: "286ac2d9-4dff-4b4c-8db2-78e0a944c990"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:18:03.054003 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:03.053972    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286ac2d9-4dff-4b4c-8db2-78e0a944c990-kube-api-access-j98hn" (OuterVolumeSpecName: "kube-api-access-j98hn") pod "286ac2d9-4dff-4b4c-8db2-78e0a944c990" (UID: "286ac2d9-4dff-4b4c-8db2-78e0a944c990"). InnerVolumeSpecName "kube-api-access-j98hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:18:03.152674 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:03.152642    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j98hn\" (UniqueName: \"kubernetes.io/projected/286ac2d9-4dff-4b4c-8db2-78e0a944c990-kube-api-access-j98hn\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:18:03.152674 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:03.152668    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/286ac2d9-4dff-4b4c-8db2-78e0a944c990-plugins-config-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:18:03.590594 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:03.590561    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af0d5199-463b-4127-97c3-f68c81c9b7f7" path="/var/lib/kubelet/pods/af0d5199-463b-4127-97c3-f68c81c9b7f7/volumes"
Sep 12 01:18:03.827935 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:03.827885    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc" event={"ID":"286ac2d9-4dff-4b4c-8db2-78e0a944c990","Type":"ContainerDied","Data":"1534a9b239b45355048d2287c2b1db3cf3ffbf6c7898041edde9a7595f49d6f2"}
Sep 12 01:18:03.828406 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:03.827958    2579 scope.go:117] "RemoveContainer" containerID="2b39f99367d1254bdbb3ab8a64a2af008ab298990137549380ea5b60e9df5589"
Sep 12 01:18:03.828406 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:03.827913    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"
Sep 12 01:18:03.844260 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:03.844207    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"]
Sep 12 01:18:03.847522 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:03.847498    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ef5997-7cbccc4f9d-9c8fc"]
Sep 12 01:18:05.590025 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:05.589992    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="286ac2d9-4dff-4b4c-8db2-78e0a944c990" path="/var/lib/kubelet/pods/286ac2d9-4dff-4b4c-8db2-78e0a944c990/volumes"
Sep 12 01:18:11.880125 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:11.880090    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f459795f7-k2p55"]
Sep 12 01:18:11.880714 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:11.880285    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5f459795f7-k2p55" podUID="215d437d-6ae1-4225-9d31-a6421a20a66a" containerName="manager" containerID="cri-o://5a66eeadab7cce9f9fc83e7f27002e7a291b5059cfd1ac82e8335e4fb67cb7ed" gracePeriod=30
Sep 12 01:18:12.623073 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.623052    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 01:18:12.720293 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.720261    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv7nv\" (UniqueName: \"kubernetes.io/projected/215d437d-6ae1-4225-9d31-a6421a20a66a-kube-api-access-wv7nv\") pod \"215d437d-6ae1-4225-9d31-a6421a20a66a\" (UID: \"215d437d-6ae1-4225-9d31-a6421a20a66a\") "
Sep 12 01:18:12.720473 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.720323    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/215d437d-6ae1-4225-9d31-a6421a20a66a-metrics-certs\") pod \"215d437d-6ae1-4225-9d31-a6421a20a66a\" (UID: \"215d437d-6ae1-4225-9d31-a6421a20a66a\") "
Sep 12 01:18:12.720473 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.720372    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/215d437d-6ae1-4225-9d31-a6421a20a66a-webhook-cert\") pod \"215d437d-6ae1-4225-9d31-a6421a20a66a\" (UID: \"215d437d-6ae1-4225-9d31-a6421a20a66a\") "
Sep 12 01:18:12.722456 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.722422    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215d437d-6ae1-4225-9d31-a6421a20a66a-kube-api-access-wv7nv" (OuterVolumeSpecName: "kube-api-access-wv7nv") pod "215d437d-6ae1-4225-9d31-a6421a20a66a" (UID: "215d437d-6ae1-4225-9d31-a6421a20a66a"). InnerVolumeSpecName "kube-api-access-wv7nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:18:12.722566 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.722448    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/215d437d-6ae1-4225-9d31-a6421a20a66a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "215d437d-6ae1-4225-9d31-a6421a20a66a" (UID: "215d437d-6ae1-4225-9d31-a6421a20a66a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:18:12.722566 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.722467    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/215d437d-6ae1-4225-9d31-a6421a20a66a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "215d437d-6ae1-4225-9d31-a6421a20a66a" (UID: "215d437d-6ae1-4225-9d31-a6421a20a66a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:18:12.821629 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.821545    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv7nv\" (UniqueName: \"kubernetes.io/projected/215d437d-6ae1-4225-9d31-a6421a20a66a-kube-api-access-wv7nv\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:18:12.821629 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.821571    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/215d437d-6ae1-4225-9d31-a6421a20a66a-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:18:12.821629 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.821580    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/215d437d-6ae1-4225-9d31-a6421a20a66a-webhook-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:18:12.868633 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.868588    2579 generic.go:356] "Generic (PLEG): container finished" podID="215d437d-6ae1-4225-9d31-a6421a20a66a" containerID="5a66eeadab7cce9f9fc83e7f27002e7a291b5059cfd1ac82e8335e4fb67cb7ed" exitCode=0
Sep 12 01:18:12.868760 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.868665    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f459795f7-k2p55"
Sep 12 01:18:12.868760 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.868672    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f459795f7-k2p55" event={"ID":"215d437d-6ae1-4225-9d31-a6421a20a66a","Type":"ContainerDied","Data":"5a66eeadab7cce9f9fc83e7f27002e7a291b5059cfd1ac82e8335e4fb67cb7ed"}
Sep 12 01:18:12.868760 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.868752    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f459795f7-k2p55" event={"ID":"215d437d-6ae1-4225-9d31-a6421a20a66a","Type":"ContainerDied","Data":"c854f4c2d59e585f0d51093c4f8e789600204a512a17a59933323d14af6f71fa"}
Sep 12 01:18:12.868887 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.868776    2579 scope.go:117] "RemoveContainer" containerID="5a66eeadab7cce9f9fc83e7f27002e7a291b5059cfd1ac82e8335e4fb67cb7ed"
Sep 12 01:18:12.878084 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.878065    2579 scope.go:117] "RemoveContainer" containerID="5a66eeadab7cce9f9fc83e7f27002e7a291b5059cfd1ac82e8335e4fb67cb7ed"
Sep 12 01:18:12.878358 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:18:12.878340    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a66eeadab7cce9f9fc83e7f27002e7a291b5059cfd1ac82e8335e4fb67cb7ed\": container with ID starting with 5a66eeadab7cce9f9fc83e7f27002e7a291b5059cfd1ac82e8335e4fb67cb7ed not found: ID does not exist" containerID="5a66eeadab7cce9f9fc83e7f27002e7a291b5059cfd1ac82e8335e4fb67cb7ed"
Sep 12 01:18:12.878414 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.878366    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a66eeadab7cce9f9fc83e7f27002e7a291b5059cfd1ac82e8335e4fb67cb7ed"} err="failed to get container status \"5a66eeadab7cce9f9fc83e7f27002e7a291b5059cfd1ac82e8335e4fb67cb7ed\": rpc error: code = NotFound desc = could not find container \"5a66eeadab7cce9f9fc83e7f27002e7a291b5059cfd1ac82e8335e4fb67cb7ed\": container with ID starting with 5a66eeadab7cce9f9fc83e7f27002e7a291b5059cfd1ac82e8335e4fb67cb7ed not found: ID does not exist"
Sep 12 01:18:12.891023 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.891002    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f459795f7-k2p55"]
Sep 12 01:18:12.895050 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:12.895030    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f459795f7-k2p55"]
Sep 12 01:18:13.591120 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:18:13.591087    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="215d437d-6ae1-4225-9d31-a6421a20a66a" path="/var/lib/kubelet/pods/215d437d-6ae1-4225-9d31-a6421a20a66a/volumes"
Sep 12 01:19:13.716986 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.716950    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f459795f7-tsgft"]
Sep 12 01:19:13.717395 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.717247    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="286ac2d9-4dff-4b4c-8db2-78e0a944c990" containerName="payload-pre-processing"
Sep 12 01:19:13.717395 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.717260    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="286ac2d9-4dff-4b4c-8db2-78e0a944c990" containerName="payload-pre-processing"
Sep 12 01:19:13.717395 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.717267    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="215d437d-6ae1-4225-9d31-a6421a20a66a" containerName="manager"
Sep 12 01:19:13.717395 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.717272    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="215d437d-6ae1-4225-9d31-a6421a20a66a" containerName="manager"
Sep 12 01:19:13.717395 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.717285    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af0d5199-463b-4127-97c3-f68c81c9b7f7" containerName="revoke-keys"
Sep 12 01:19:13.717395 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.717290    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="af0d5199-463b-4127-97c3-f68c81c9b7f7" containerName="revoke-keys"
Sep 12 01:19:13.717395 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.717338    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="286ac2d9-4dff-4b4c-8db2-78e0a944c990" containerName="payload-pre-processing"
Sep 12 01:19:13.717395 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.717345    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="215d437d-6ae1-4225-9d31-a6421a20a66a" containerName="manager"
Sep 12 01:19:13.717395 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.717355    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="af0d5199-463b-4127-97c3-f68c81c9b7f7" containerName="revoke-keys"
Sep 12 01:19:13.720321 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.720301    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f459795f7-tsgft"
Sep 12 01:19:13.723784 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.723762    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-484sv\""
Sep 12 01:19:13.723784 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.723780    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 12 01:19:13.723926 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.723779    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 12 01:19:13.723926 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.723762    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 12 01:19:13.730766 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.730745    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f459795f7-tsgft"]
Sep 12 01:19:13.881966 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.881928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6912a0ce-bee3-4689-b539-bbcc8917a4a0-webhook-cert\") pod \"maas-controller-5f459795f7-tsgft\" (UID: \"6912a0ce-bee3-4689-b539-bbcc8917a4a0\") " pod="opendatahub/maas-controller-5f459795f7-tsgft"
Sep 12 01:19:13.882164 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.881972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6912a0ce-bee3-4689-b539-bbcc8917a4a0-metrics-certs\") pod \"maas-controller-5f459795f7-tsgft\" (UID: \"6912a0ce-bee3-4689-b539-bbcc8917a4a0\") " pod="opendatahub/maas-controller-5f459795f7-tsgft"
Sep 12 01:19:13.882164 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.882041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgbh2\" (UniqueName: \"kubernetes.io/projected/6912a0ce-bee3-4689-b539-bbcc8917a4a0-kube-api-access-zgbh2\") pod \"maas-controller-5f459795f7-tsgft\" (UID: \"6912a0ce-bee3-4689-b539-bbcc8917a4a0\") " pod="opendatahub/maas-controller-5f459795f7-tsgft"
Sep 12 01:19:13.983081 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.982990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6912a0ce-bee3-4689-b539-bbcc8917a4a0-webhook-cert\") pod \"maas-controller-5f459795f7-tsgft\" (UID: \"6912a0ce-bee3-4689-b539-bbcc8917a4a0\") " pod="opendatahub/maas-controller-5f459795f7-tsgft"
Sep 12 01:19:13.983081 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.983033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6912a0ce-bee3-4689-b539-bbcc8917a4a0-metrics-certs\") pod \"maas-controller-5f459795f7-tsgft\" (UID: \"6912a0ce-bee3-4689-b539-bbcc8917a4a0\") " pod="opendatahub/maas-controller-5f459795f7-tsgft"
Sep 12 01:19:13.983323 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.983086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgbh2\" (UniqueName: \"kubernetes.io/projected/6912a0ce-bee3-4689-b539-bbcc8917a4a0-kube-api-access-zgbh2\") pod \"maas-controller-5f459795f7-tsgft\" (UID: \"6912a0ce-bee3-4689-b539-bbcc8917a4a0\") " pod="opendatahub/maas-controller-5f459795f7-tsgft"
Sep 12 01:19:13.985509 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.985481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6912a0ce-bee3-4689-b539-bbcc8917a4a0-webhook-cert\") pod \"maas-controller-5f459795f7-tsgft\" (UID: \"6912a0ce-bee3-4689-b539-bbcc8917a4a0\") " pod="opendatahub/maas-controller-5f459795f7-tsgft"
Sep 12 01:19:13.985648 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.985624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6912a0ce-bee3-4689-b539-bbcc8917a4a0-metrics-certs\") pod \"maas-controller-5f459795f7-tsgft\" (UID: \"6912a0ce-bee3-4689-b539-bbcc8917a4a0\") " pod="opendatahub/maas-controller-5f459795f7-tsgft"
Sep 12 01:19:13.991429 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:13.991399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgbh2\" (UniqueName: \"kubernetes.io/projected/6912a0ce-bee3-4689-b539-bbcc8917a4a0-kube-api-access-zgbh2\") pod \"maas-controller-5f459795f7-tsgft\" (UID: \"6912a0ce-bee3-4689-b539-bbcc8917a4a0\") " pod="opendatahub/maas-controller-5f459795f7-tsgft"
Sep 12 01:19:14.030393 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:14.030360    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f459795f7-tsgft"
Sep 12 01:19:14.304308 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:14.304277    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f459795f7-tsgft"]
Sep 12 01:19:14.305910 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:19:14.305885    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6912a0ce_bee3_4689_b539_bbcc8917a4a0.slice/crio-b4fcbc69aab228d0945966d65288821f84dc8270299fee91497ce49d2be04fea WatchSource:0}: Error finding container b4fcbc69aab228d0945966d65288821f84dc8270299fee91497ce49d2be04fea: Status 404 returned error can't find the container with id b4fcbc69aab228d0945966d65288821f84dc8270299fee91497ce49d2be04fea
Sep 12 01:19:14.307209 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:14.307188    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 01:19:15.158970 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:15.158880    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f459795f7-tsgft" event={"ID":"6912a0ce-bee3-4689-b539-bbcc8917a4a0","Type":"ContainerStarted","Data":"e5aea7b6aea0bcece1eb2a81b7e593285f178e7db02b4e37df8436bf7e7b2870"}
Sep 12 01:19:15.158970 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:15.158931    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f459795f7-tsgft" event={"ID":"6912a0ce-bee3-4689-b539-bbcc8917a4a0","Type":"ContainerStarted","Data":"b4fcbc69aab228d0945966d65288821f84dc8270299fee91497ce49d2be04fea"}
Sep 12 01:19:15.159446 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:15.159019    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f459795f7-tsgft"
Sep 12 01:19:15.177807 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:15.177762    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5f459795f7-tsgft" podStartSLOduration=1.734368189 podStartE2EDuration="2.1777491s" totalImagesPullingTime="443.380911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:19:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:19:15.176122501 +0000 UTC m=+2180.095414289" watchObservedRunningTime="2026-09-12 01:19:15.1777491 +0000 UTC m=+2180.097040886"
Sep 12 01:19:26.165151 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:26.165120    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f459795f7-tsgft"
Sep 12 01:19:33.441957 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:33.441917    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f459795f7-tsgft"]
Sep 12 01:19:33.442368 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:33.442163    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5f459795f7-tsgft" podUID="6912a0ce-bee3-4689-b539-bbcc8917a4a0" containerName="manager" containerID="cri-o://e5aea7b6aea0bcece1eb2a81b7e593285f178e7db02b4e37df8436bf7e7b2870" gracePeriod=30
Sep 12 01:19:33.684977 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:33.684951    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f459795f7-tsgft"
Sep 12 01:19:33.730906 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:33.730838    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6912a0ce-bee3-4689-b539-bbcc8917a4a0-metrics-certs\") pod \"6912a0ce-bee3-4689-b539-bbcc8917a4a0\" (UID: \"6912a0ce-bee3-4689-b539-bbcc8917a4a0\") "
Sep 12 01:19:33.730906 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:33.730871    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgbh2\" (UniqueName: \"kubernetes.io/projected/6912a0ce-bee3-4689-b539-bbcc8917a4a0-kube-api-access-zgbh2\") pod \"6912a0ce-bee3-4689-b539-bbcc8917a4a0\" (UID: \"6912a0ce-bee3-4689-b539-bbcc8917a4a0\") "
Sep 12 01:19:33.730906 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:33.730906    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6912a0ce-bee3-4689-b539-bbcc8917a4a0-webhook-cert\") pod \"6912a0ce-bee3-4689-b539-bbcc8917a4a0\" (UID: \"6912a0ce-bee3-4689-b539-bbcc8917a4a0\") "
Sep 12 01:19:33.732913 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:33.732880    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6912a0ce-bee3-4689-b539-bbcc8917a4a0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6912a0ce-bee3-4689-b539-bbcc8917a4a0" (UID: "6912a0ce-bee3-4689-b539-bbcc8917a4a0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:19:33.732913 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:33.732893    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6912a0ce-bee3-4689-b539-bbcc8917a4a0-kube-api-access-zgbh2" (OuterVolumeSpecName: "kube-api-access-zgbh2") pod "6912a0ce-bee3-4689-b539-bbcc8917a4a0" (UID: "6912a0ce-bee3-4689-b539-bbcc8917a4a0"). InnerVolumeSpecName "kube-api-access-zgbh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:19:33.733069 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:33.732980    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6912a0ce-bee3-4689-b539-bbcc8917a4a0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6912a0ce-bee3-4689-b539-bbcc8917a4a0" (UID: "6912a0ce-bee3-4689-b539-bbcc8917a4a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:19:33.831345 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:33.831309    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6912a0ce-bee3-4689-b539-bbcc8917a4a0-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:19:33.831345 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:33.831337    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgbh2\" (UniqueName: \"kubernetes.io/projected/6912a0ce-bee3-4689-b539-bbcc8917a4a0-kube-api-access-zgbh2\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:19:33.831345 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:33.831350    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6912a0ce-bee3-4689-b539-bbcc8917a4a0-webhook-cert\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:19:34.244120 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:34.244085    2579 generic.go:356] "Generic (PLEG): container finished" podID="6912a0ce-bee3-4689-b539-bbcc8917a4a0" containerID="e5aea7b6aea0bcece1eb2a81b7e593285f178e7db02b4e37df8436bf7e7b2870" exitCode=0
Sep 12 01:19:34.244266 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:34.244135    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f459795f7-tsgft"
Sep 12 01:19:34.244266 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:34.244149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f459795f7-tsgft" event={"ID":"6912a0ce-bee3-4689-b539-bbcc8917a4a0","Type":"ContainerDied","Data":"e5aea7b6aea0bcece1eb2a81b7e593285f178e7db02b4e37df8436bf7e7b2870"}
Sep 12 01:19:34.244266 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:34.244201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f459795f7-tsgft" event={"ID":"6912a0ce-bee3-4689-b539-bbcc8917a4a0","Type":"ContainerDied","Data":"b4fcbc69aab228d0945966d65288821f84dc8270299fee91497ce49d2be04fea"}
Sep 12 01:19:34.244266 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:34.244232    2579 scope.go:117] "RemoveContainer" containerID="e5aea7b6aea0bcece1eb2a81b7e593285f178e7db02b4e37df8436bf7e7b2870"
Sep 12 01:19:34.253025 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:34.253006    2579 scope.go:117] "RemoveContainer" containerID="e5aea7b6aea0bcece1eb2a81b7e593285f178e7db02b4e37df8436bf7e7b2870"
Sep 12 01:19:34.253275 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:19:34.253255    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5aea7b6aea0bcece1eb2a81b7e593285f178e7db02b4e37df8436bf7e7b2870\": container with ID starting with e5aea7b6aea0bcece1eb2a81b7e593285f178e7db02b4e37df8436bf7e7b2870 not found: ID does not exist" containerID="e5aea7b6aea0bcece1eb2a81b7e593285f178e7db02b4e37df8436bf7e7b2870"
Sep 12 01:19:34.253346 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:34.253282    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5aea7b6aea0bcece1eb2a81b7e593285f178e7db02b4e37df8436bf7e7b2870"} err="failed to get container status \"e5aea7b6aea0bcece1eb2a81b7e593285f178e7db02b4e37df8436bf7e7b2870\": rpc error: code = NotFound desc = could not find container \"e5aea7b6aea0bcece1eb2a81b7e593285f178e7db02b4e37df8436bf7e7b2870\": container with ID starting with e5aea7b6aea0bcece1eb2a81b7e593285f178e7db02b4e37df8436bf7e7b2870 not found: ID does not exist"
Sep 12 01:19:34.266642 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:34.266622    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f459795f7-tsgft"]
Sep 12 01:19:34.272060 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:34.272040    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f459795f7-tsgft"]
Sep 12 01:19:35.106172 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.106128    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f459795f7-ff5sd"]
Sep 12 01:19:35.106686 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.106581    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6912a0ce-bee3-4689-b539-bbcc8917a4a0" containerName="manager"
Sep 12 01:19:35.106686 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.106625    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6912a0ce-bee3-4689-b539-bbcc8917a4a0" containerName="manager"
Sep 12 01:19:35.106834 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.106715    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6912a0ce-bee3-4689-b539-bbcc8917a4a0" containerName="manager"
Sep 12 01:19:35.111173 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.111149    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f459795f7-ff5sd"
Sep 12 01:19:35.115108 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.114865    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 12 01:19:35.115108 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.114919    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-484sv\""
Sep 12 01:19:35.115108 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.114931    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 12 01:19:35.115392 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.115374    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 12 01:19:35.121813 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.121790    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f459795f7-ff5sd"]
Sep 12 01:19:35.143359 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.143329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm7zs\" (UniqueName: \"kubernetes.io/projected/d6ca53e2-99c0-4105-8d23-6a0e4b0329e7-kube-api-access-lm7zs\") pod \"maas-controller-5f459795f7-ff5sd\" (UID: \"d6ca53e2-99c0-4105-8d23-6a0e4b0329e7\") " pod="opendatahub/maas-controller-5f459795f7-ff5sd"
Sep 12 01:19:35.143475 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.143378    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6ca53e2-99c0-4105-8d23-6a0e4b0329e7-metrics-certs\") pod \"maas-controller-5f459795f7-ff5sd\" (UID: \"d6ca53e2-99c0-4105-8d23-6a0e4b0329e7\") " pod="opendatahub/maas-controller-5f459795f7-ff5sd"
Sep 12 01:19:35.143533 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.143493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6ca53e2-99c0-4105-8d23-6a0e4b0329e7-webhook-cert\") pod \"maas-controller-5f459795f7-ff5sd\" (UID: \"d6ca53e2-99c0-4105-8d23-6a0e4b0329e7\") " pod="opendatahub/maas-controller-5f459795f7-ff5sd"
Sep 12 01:19:35.244686 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.244651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6ca53e2-99c0-4105-8d23-6a0e4b0329e7-webhook-cert\") pod \"maas-controller-5f459795f7-ff5sd\" (UID: \"d6ca53e2-99c0-4105-8d23-6a0e4b0329e7\") " pod="opendatahub/maas-controller-5f459795f7-ff5sd"
Sep 12 01:19:35.244819 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.244737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm7zs\" (UniqueName: \"kubernetes.io/projected/d6ca53e2-99c0-4105-8d23-6a0e4b0329e7-kube-api-access-lm7zs\") pod \"maas-controller-5f459795f7-ff5sd\" (UID: \"d6ca53e2-99c0-4105-8d23-6a0e4b0329e7\") " pod="opendatahub/maas-controller-5f459795f7-ff5sd"
Sep 12 01:19:35.244819 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.244756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6ca53e2-99c0-4105-8d23-6a0e4b0329e7-metrics-certs\") pod \"maas-controller-5f459795f7-ff5sd\" (UID: \"d6ca53e2-99c0-4105-8d23-6a0e4b0329e7\") " pod="opendatahub/maas-controller-5f459795f7-ff5sd"
Sep 12 01:19:35.247120 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.247089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d6ca53e2-99c0-4105-8d23-6a0e4b0329e7-webhook-cert\") pod \"maas-controller-5f459795f7-ff5sd\" (UID: \"d6ca53e2-99c0-4105-8d23-6a0e4b0329e7\") " pod="opendatahub/maas-controller-5f459795f7-ff5sd"
Sep 12 01:19:35.247225 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.247166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6ca53e2-99c0-4105-8d23-6a0e4b0329e7-metrics-certs\") pod \"maas-controller-5f459795f7-ff5sd\" (UID: \"d6ca53e2-99c0-4105-8d23-6a0e4b0329e7\") " pod="opendatahub/maas-controller-5f459795f7-ff5sd"
Sep 12 01:19:35.253153 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.253130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm7zs\" (UniqueName: \"kubernetes.io/projected/d6ca53e2-99c0-4105-8d23-6a0e4b0329e7-kube-api-access-lm7zs\") pod \"maas-controller-5f459795f7-ff5sd\" (UID: \"d6ca53e2-99c0-4105-8d23-6a0e4b0329e7\") " pod="opendatahub/maas-controller-5f459795f7-ff5sd"
Sep 12 01:19:35.426642 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.426617    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f459795f7-ff5sd"
Sep 12 01:19:35.591398 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.591368    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6912a0ce-bee3-4689-b539-bbcc8917a4a0" path="/var/lib/kubelet/pods/6912a0ce-bee3-4689-b539-bbcc8917a4a0/volumes"
Sep 12 01:19:35.699099 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:35.699044    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f459795f7-ff5sd"]
Sep 12 01:19:35.701587 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:19:35.701560    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ca53e2_99c0_4105_8d23_6a0e4b0329e7.slice/crio-d1e95fecd41efbe97d07ab597357f7d85cd1f7570102403a13a27d696bca1edf WatchSource:0}: Error finding container d1e95fecd41efbe97d07ab597357f7d85cd1f7570102403a13a27d696bca1edf: Status 404 returned error can't find the container with id d1e95fecd41efbe97d07ab597357f7d85cd1f7570102403a13a27d696bca1edf
Sep 12 01:19:36.256402 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:36.256368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f459795f7-ff5sd" event={"ID":"d6ca53e2-99c0-4105-8d23-6a0e4b0329e7","Type":"ContainerStarted","Data":"376679a2f85b062a20a5891909d7d08b2444fb0c75b23ead4444db1c68753508"}
Sep 12 01:19:36.256870 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:36.256430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f459795f7-ff5sd" event={"ID":"d6ca53e2-99c0-4105-8d23-6a0e4b0329e7","Type":"ContainerStarted","Data":"d1e95fecd41efbe97d07ab597357f7d85cd1f7570102403a13a27d696bca1edf"}
Sep 12 01:19:36.256870 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:36.256486    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f459795f7-ff5sd"
Sep 12 01:19:36.276009 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:36.275911    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5f459795f7-ff5sd" podStartSLOduration=0.96925914 podStartE2EDuration="1.275896945s" totalImagesPullingTime="306.637805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:19:36.274182051 +0000 UTC m=+2201.193473840" watchObservedRunningTime="2026-09-12 01:19:36.275896945 +0000 UTC m=+2201.195188732"
Sep 12 01:19:47.268092 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:19:47.268064    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f459795f7-ff5sd"
Sep 12 01:20:43.288636 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:43.288587    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"]
Sep 12 01:20:43.289120 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:43.288805    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66" podUID="1a4985c0-2f5f-4392-a8d5-a0eaaf913444" containerName="manager" containerID="cri-o://66bd9f00bb396479c3b579a1f63f5aca349e49cf7e1ea31f371113819eb0024f" gracePeriod=10
Sep 12 01:20:45.331963 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.331937    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"
Sep 12 01:20:45.383786 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.383705    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a4985c0-2f5f-4392-a8d5-a0eaaf913444-extensions-socket-volume\") pod \"1a4985c0-2f5f-4392-a8d5-a0eaaf913444\" (UID: \"1a4985c0-2f5f-4392-a8d5-a0eaaf913444\") "
Sep 12 01:20:45.383786 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.383748    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4txd7\" (UniqueName: \"kubernetes.io/projected/1a4985c0-2f5f-4392-a8d5-a0eaaf913444-kube-api-access-4txd7\") pod \"1a4985c0-2f5f-4392-a8d5-a0eaaf913444\" (UID: \"1a4985c0-2f5f-4392-a8d5-a0eaaf913444\") "
Sep 12 01:20:45.384111 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.384089    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4985c0-2f5f-4392-a8d5-a0eaaf913444-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1a4985c0-2f5f-4392-a8d5-a0eaaf913444" (UID: "1a4985c0-2f5f-4392-a8d5-a0eaaf913444"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:20:45.385745 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.385721    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4985c0-2f5f-4392-a8d5-a0eaaf913444-kube-api-access-4txd7" (OuterVolumeSpecName: "kube-api-access-4txd7") pod "1a4985c0-2f5f-4392-a8d5-a0eaaf913444" (UID: "1a4985c0-2f5f-4392-a8d5-a0eaaf913444"). InnerVolumeSpecName "kube-api-access-4txd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:20:45.484327 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.484296    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a4985c0-2f5f-4392-a8d5-a0eaaf913444-extensions-socket-volume\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:20:45.484327 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.484320    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4txd7\" (UniqueName: \"kubernetes.io/projected/1a4985c0-2f5f-4392-a8d5-a0eaaf913444-kube-api-access-4txd7\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:20:45.575571 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.575541    2579 generic.go:356] "Generic (PLEG): container finished" podID="1a4985c0-2f5f-4392-a8d5-a0eaaf913444" containerID="66bd9f00bb396479c3b579a1f63f5aca349e49cf7e1ea31f371113819eb0024f" exitCode=0
Sep 12 01:20:45.575711 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.575618    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"
Sep 12 01:20:45.575711 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.575624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66" event={"ID":"1a4985c0-2f5f-4392-a8d5-a0eaaf913444","Type":"ContainerDied","Data":"66bd9f00bb396479c3b579a1f63f5aca349e49cf7e1ea31f371113819eb0024f"}
Sep 12 01:20:45.575711 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.575683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66" event={"ID":"1a4985c0-2f5f-4392-a8d5-a0eaaf913444","Type":"ContainerDied","Data":"255bed2e4e0d3bfa75beeb0c99a5b63cf7355a77e6608545539c6780e3cd520d"}
Sep 12 01:20:45.575711 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.575706    2579 scope.go:117] "RemoveContainer" containerID="66bd9f00bb396479c3b579a1f63f5aca349e49cf7e1ea31f371113819eb0024f"
Sep 12 01:20:45.584182 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.584160    2579 scope.go:117] "RemoveContainer" containerID="32eac081121d4e8724e6365b5402f453490a76341a062f6493fcd4d12c124215"
Sep 12 01:20:45.593455 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.593437    2579 scope.go:117] "RemoveContainer" containerID="66bd9f00bb396479c3b579a1f63f5aca349e49cf7e1ea31f371113819eb0024f"
Sep 12 01:20:45.593784 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:20:45.593763    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66bd9f00bb396479c3b579a1f63f5aca349e49cf7e1ea31f371113819eb0024f\": container with ID starting with 66bd9f00bb396479c3b579a1f63f5aca349e49cf7e1ea31f371113819eb0024f not found: ID does not exist" containerID="66bd9f00bb396479c3b579a1f63f5aca349e49cf7e1ea31f371113819eb0024f"
Sep 12 01:20:45.593861 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.593793    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66bd9f00bb396479c3b579a1f63f5aca349e49cf7e1ea31f371113819eb0024f"} err="failed to get container status \"66bd9f00bb396479c3b579a1f63f5aca349e49cf7e1ea31f371113819eb0024f\": rpc error: code = NotFound desc = could not find container \"66bd9f00bb396479c3b579a1f63f5aca349e49cf7e1ea31f371113819eb0024f\": container with ID starting with 66bd9f00bb396479c3b579a1f63f5aca349e49cf7e1ea31f371113819eb0024f not found: ID does not exist"
Sep 12 01:20:45.593861 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.593809    2579 scope.go:117] "RemoveContainer" containerID="32eac081121d4e8724e6365b5402f453490a76341a062f6493fcd4d12c124215"
Sep 12 01:20:45.594072 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:20:45.594057    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32eac081121d4e8724e6365b5402f453490a76341a062f6493fcd4d12c124215\": container with ID starting with 32eac081121d4e8724e6365b5402f453490a76341a062f6493fcd4d12c124215 not found: ID does not exist" containerID="32eac081121d4e8724e6365b5402f453490a76341a062f6493fcd4d12c124215"
Sep 12 01:20:45.594119 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.594076    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32eac081121d4e8724e6365b5402f453490a76341a062f6493fcd4d12c124215"} err="failed to get container status \"32eac081121d4e8724e6365b5402f453490a76341a062f6493fcd4d12c124215\": rpc error: code = NotFound desc = could not find container \"32eac081121d4e8724e6365b5402f453490a76341a062f6493fcd4d12c124215\": container with ID starting with 32eac081121d4e8724e6365b5402f453490a76341a062f6493fcd4d12c124215 not found: ID does not exist"
Sep 12 01:20:45.597649 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.597624    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"]
Sep 12 01:20:45.602899 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:45.602877    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hrv66"]
Sep 12 01:20:47.590250 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:20:47.590222    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a4985c0-2f5f-4392-a8d5-a0eaaf913444" path="/var/lib/kubelet/pods/1a4985c0-2f5f-4392-a8d5-a0eaaf913444/volumes"
Sep 12 01:21:43.995784 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:43.995749    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4"]
Sep 12 01:21:43.996222 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:43.996039    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a4985c0-2f5f-4392-a8d5-a0eaaf913444" containerName="manager"
Sep 12 01:21:43.996222 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:43.996051    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a4985c0-2f5f-4392-a8d5-a0eaaf913444" containerName="manager"
Sep 12 01:21:43.996222 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:43.996065    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a4985c0-2f5f-4392-a8d5-a0eaaf913444" containerName="manager"
Sep 12 01:21:43.996222 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:43.996071    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a4985c0-2f5f-4392-a8d5-a0eaaf913444" containerName="manager"
Sep 12 01:21:43.996222 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:43.996129    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a4985c0-2f5f-4392-a8d5-a0eaaf913444" containerName="manager"
Sep 12 01:21:43.996222 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:43.996136    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a4985c0-2f5f-4392-a8d5-a0eaaf913444" containerName="manager"
Sep 12 01:21:43.998865 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:43.998848    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4"
Sep 12 01:21:44.001937 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:44.001916    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8nhgh\""
Sep 12 01:21:44.012560 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:44.012538    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4"]
Sep 12 01:21:44.131882 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:44.131841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8a9200b-2ad5-41f7-8cb4-81e1cedfb143-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cnzl4\" (UID: \"a8a9200b-2ad5-41f7-8cb4-81e1cedfb143\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4"
Sep 12 01:21:44.132063 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:44.131924    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5g5j\" (UniqueName: \"kubernetes.io/projected/a8a9200b-2ad5-41f7-8cb4-81e1cedfb143-kube-api-access-p5g5j\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cnzl4\" (UID: \"a8a9200b-2ad5-41f7-8cb4-81e1cedfb143\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4"
Sep 12 01:21:44.232894 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:44.232853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5g5j\" (UniqueName: \"kubernetes.io/projected/a8a9200b-2ad5-41f7-8cb4-81e1cedfb143-kube-api-access-p5g5j\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cnzl4\" (UID: \"a8a9200b-2ad5-41f7-8cb4-81e1cedfb143\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4"
Sep 12 01:21:44.233052 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:44.232941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8a9200b-2ad5-41f7-8cb4-81e1cedfb143-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cnzl4\" (UID: \"a8a9200b-2ad5-41f7-8cb4-81e1cedfb143\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4"
Sep 12 01:21:44.233338 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:44.233318    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8a9200b-2ad5-41f7-8cb4-81e1cedfb143-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cnzl4\" (UID: \"a8a9200b-2ad5-41f7-8cb4-81e1cedfb143\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4"
Sep 12 01:21:44.242553 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:44.242532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5g5j\" (UniqueName: \"kubernetes.io/projected/a8a9200b-2ad5-41f7-8cb4-81e1cedfb143-kube-api-access-p5g5j\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cnzl4\" (UID: \"a8a9200b-2ad5-41f7-8cb4-81e1cedfb143\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4"
Sep 12 01:21:44.308536 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:44.308459    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4"
Sep 12 01:21:44.585530 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:44.585504    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4"]
Sep 12 01:21:44.587245 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:21:44.587221    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a9200b_2ad5_41f7_8cb4_81e1cedfb143.slice/crio-9510479a21742a131d363f46c2882b3d44be2c64bee145988096d563c9c3e589 WatchSource:0}: Error finding container 9510479a21742a131d363f46c2882b3d44be2c64bee145988096d563c9c3e589: Status 404 returned error can't find the container with id 9510479a21742a131d363f46c2882b3d44be2c64bee145988096d563c9c3e589
Sep 12 01:21:44.834811 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:44.834718    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4" event={"ID":"a8a9200b-2ad5-41f7-8cb4-81e1cedfb143","Type":"ContainerStarted","Data":"895ca1a1e80acffd63b295cc4baa76a79b1a8edd519dee212327b9459535c70f"}
Sep 12 01:21:44.834967 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:44.834811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4" event={"ID":"a8a9200b-2ad5-41f7-8cb4-81e1cedfb143","Type":"ContainerStarted","Data":"9510479a21742a131d363f46c2882b3d44be2c64bee145988096d563c9c3e589"}
Sep 12 01:21:44.834967 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:44.834856    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4"
Sep 12 01:21:44.885565 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:44.885510    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4" podStartSLOduration=1.885496357 podStartE2EDuration="1.885496357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:21:44.883498485 +0000 UTC m=+2329.802790284" watchObservedRunningTime="2026-09-12 01:21:44.885496357 +0000 UTC m=+2329.804788144"
Sep 12 01:21:55.842237 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:55.842160    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnzl4"
Sep 12 01:21:56.029208 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:21:56.029180    2579 scope.go:117] "RemoveContainer" containerID="40b415c059e9fd0f79f73b69750c4c2da8a0a23c1a18a354969917415f189113"
Sep 12 01:22:55.695563 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:22:55.695532    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 01:22:55.699846 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:22:55.699826    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qzkgb_45c04ace-43ff-47a5-a953-68957b92aba4/ovn-acl-logging/0.log"
Sep 12 01:22:56.046915 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:22:56.046834    2579 scope.go:117] "RemoveContainer" containerID="9cb7486ae2e1127b5e2be83a8923bc3604f61851e59cf358b5b34a1158fda424"
Sep 12 01:23:29.371430 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.371394    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"]
Sep 12 01:23:29.374809 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.374790    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:29.378738 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.378715    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 12 01:23:29.378898 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.378771    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 12 01:23:29.378898 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.378805    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 12 01:23:29.378898 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.378851    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k2gll\""
Sep 12 01:23:29.379041 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.378778    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:23:29.387049 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.387027    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"]
Sep 12 01:23:29.450325 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.450288    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1d0d273-9a23-425b-b389-0a38ca5fcc52-metrics-certs\") pod \"maas-api-6cc46ff6dc-dkxj2\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:29.450325 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.450328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l944s\" (UniqueName: \"kubernetes.io/projected/c1d0d273-9a23-425b-b389-0a38ca5fcc52-kube-api-access-l944s\") pod \"maas-api-6cc46ff6dc-dkxj2\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:29.450509 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.450378    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d0d273-9a23-425b-b389-0a38ca5fcc52-maas-api-tls\") pod \"maas-api-6cc46ff6dc-dkxj2\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:29.450509 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.450413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d0d273-9a23-425b-b389-0a38ca5fcc52-openshift-service-ca\") pod \"maas-api-6cc46ff6dc-dkxj2\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:29.550822 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.550792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1d0d273-9a23-425b-b389-0a38ca5fcc52-metrics-certs\") pod \"maas-api-6cc46ff6dc-dkxj2\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:29.550971 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.550832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l944s\" (UniqueName: \"kubernetes.io/projected/c1d0d273-9a23-425b-b389-0a38ca5fcc52-kube-api-access-l944s\") pod \"maas-api-6cc46ff6dc-dkxj2\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:29.550971 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.550866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d0d273-9a23-425b-b389-0a38ca5fcc52-maas-api-tls\") pod \"maas-api-6cc46ff6dc-dkxj2\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:29.550971 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.550890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d0d273-9a23-425b-b389-0a38ca5fcc52-openshift-service-ca\") pod \"maas-api-6cc46ff6dc-dkxj2\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:29.551577 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.551554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d0d273-9a23-425b-b389-0a38ca5fcc52-openshift-service-ca\") pod \"maas-api-6cc46ff6dc-dkxj2\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:29.553256 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.553228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d0d273-9a23-425b-b389-0a38ca5fcc52-maas-api-tls\") pod \"maas-api-6cc46ff6dc-dkxj2\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:29.553378 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.553359    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1d0d273-9a23-425b-b389-0a38ca5fcc52-metrics-certs\") pod \"maas-api-6cc46ff6dc-dkxj2\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:29.559883 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.559856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l944s\" (UniqueName: \"kubernetes.io/projected/c1d0d273-9a23-425b-b389-0a38ca5fcc52-kube-api-access-l944s\") pod \"maas-api-6cc46ff6dc-dkxj2\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") " pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:29.684752 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.684726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:29.956962 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:29.956930    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"]
Sep 12 01:23:29.958501 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:23:29.958470    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d0d273_9a23_425b_b389_0a38ca5fcc52.slice/crio-7d2623194d364c1fe5c417e68ccb9e18e13464408681b41b5cc8fee804e2e4b9 WatchSource:0}: Error finding container 7d2623194d364c1fe5c417e68ccb9e18e13464408681b41b5cc8fee804e2e4b9: Status 404 returned error can't find the container with id 7d2623194d364c1fe5c417e68ccb9e18e13464408681b41b5cc8fee804e2e4b9
Sep 12 01:23:30.325581 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:30.325481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2" event={"ID":"c1d0d273-9a23-425b-b389-0a38ca5fcc52","Type":"ContainerStarted","Data":"7d2623194d364c1fe5c417e68ccb9e18e13464408681b41b5cc8fee804e2e4b9"}
Sep 12 01:23:32.335171 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:32.335138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2" event={"ID":"c1d0d273-9a23-425b-b389-0a38ca5fcc52","Type":"ContainerStarted","Data":"4d5e7cb2a248f01460a0b5e80b0a22386033e7a555e914a62faa575829f209bf"}
Sep 12 01:23:32.335541 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:32.335209    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:32.352874 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:32.352824    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2" podStartSLOduration=1.813797289 podStartE2EDuration="3.352807232s" totalImagesPullingTime="1.539009943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:23:32.352661118 +0000 UTC m=+2437.271952907" watchObservedRunningTime="2026-09-12 01:23:32.352807232 +0000 UTC m=+2437.272099019"
Sep 12 01:23:36.873416 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:36.873377    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"]
Sep 12 01:23:36.873908 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:36.873614    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2" podUID="c1d0d273-9a23-425b-b389-0a38ca5fcc52" containerName="maas-api" containerID="cri-o://4d5e7cb2a248f01460a0b5e80b0a22386033e7a555e914a62faa575829f209bf" gracePeriod=30
Sep 12 01:23:36.878625 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:36.878582    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:36.913820 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:36.913792    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"]
Sep 12 01:23:36.919118 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:36.919089    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:36.926564 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:36.926537    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"]
Sep 12 01:23:37.018863 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.018833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b0c1ea-4465-4948-8e06-02de12faa80d-maas-api-tls\") pod \"maas-api-749bf69647-vpbrh\" (UID: \"a6b0c1ea-4465-4948-8e06-02de12faa80d\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:37.019009 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.018887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b0c1ea-4465-4948-8e06-02de12faa80d-openshift-service-ca\") pod \"maas-api-749bf69647-vpbrh\" (UID: \"a6b0c1ea-4465-4948-8e06-02de12faa80d\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:37.019009 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.018939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szqpn\" (UniqueName: \"kubernetes.io/projected/a6b0c1ea-4465-4948-8e06-02de12faa80d-kube-api-access-szqpn\") pod \"maas-api-749bf69647-vpbrh\" (UID: \"a6b0c1ea-4465-4948-8e06-02de12faa80d\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:37.019084 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.019024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b0c1ea-4465-4948-8e06-02de12faa80d-metrics-certs\") pod \"maas-api-749bf69647-vpbrh\" (UID: \"a6b0c1ea-4465-4948-8e06-02de12faa80d\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:37.112352 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.112322    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:37.119921 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.119897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b0c1ea-4465-4948-8e06-02de12faa80d-metrics-certs\") pod \"maas-api-749bf69647-vpbrh\" (UID: \"a6b0c1ea-4465-4948-8e06-02de12faa80d\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:37.120023 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.119935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b0c1ea-4465-4948-8e06-02de12faa80d-maas-api-tls\") pod \"maas-api-749bf69647-vpbrh\" (UID: \"a6b0c1ea-4465-4948-8e06-02de12faa80d\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:37.120077 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.120055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b0c1ea-4465-4948-8e06-02de12faa80d-openshift-service-ca\") pod \"maas-api-749bf69647-vpbrh\" (UID: \"a6b0c1ea-4465-4948-8e06-02de12faa80d\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:37.120113 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.120074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szqpn\" (UniqueName: \"kubernetes.io/projected/a6b0c1ea-4465-4948-8e06-02de12faa80d-kube-api-access-szqpn\") pod \"maas-api-749bf69647-vpbrh\" (UID: \"a6b0c1ea-4465-4948-8e06-02de12faa80d\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:37.120769 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.120746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b0c1ea-4465-4948-8e06-02de12faa80d-openshift-service-ca\") pod \"maas-api-749bf69647-vpbrh\" (UID: \"a6b0c1ea-4465-4948-8e06-02de12faa80d\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:37.122318 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.122291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b0c1ea-4465-4948-8e06-02de12faa80d-metrics-certs\") pod \"maas-api-749bf69647-vpbrh\" (UID: \"a6b0c1ea-4465-4948-8e06-02de12faa80d\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:37.122476 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.122324    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b0c1ea-4465-4948-8e06-02de12faa80d-maas-api-tls\") pod \"maas-api-749bf69647-vpbrh\" (UID: \"a6b0c1ea-4465-4948-8e06-02de12faa80d\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:37.128991 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.128931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szqpn\" (UniqueName: \"kubernetes.io/projected/a6b0c1ea-4465-4948-8e06-02de12faa80d-kube-api-access-szqpn\") pod \"maas-api-749bf69647-vpbrh\" (UID: \"a6b0c1ea-4465-4948-8e06-02de12faa80d\") " pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:37.220624 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.220572    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d0d273-9a23-425b-b389-0a38ca5fcc52-openshift-service-ca\") pod \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") "
Sep 12 01:23:37.220797 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.220646    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l944s\" (UniqueName: \"kubernetes.io/projected/c1d0d273-9a23-425b-b389-0a38ca5fcc52-kube-api-access-l944s\") pod \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") "
Sep 12 01:23:37.220797 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.220710    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1d0d273-9a23-425b-b389-0a38ca5fcc52-metrics-certs\") pod \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") "
Sep 12 01:23:37.220797 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.220742    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d0d273-9a23-425b-b389-0a38ca5fcc52-maas-api-tls\") pod \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\" (UID: \"c1d0d273-9a23-425b-b389-0a38ca5fcc52\") "
Sep 12 01:23:37.221046 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.221017    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d0d273-9a23-425b-b389-0a38ca5fcc52-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1d0d273-9a23-425b-b389-0a38ca5fcc52" (UID: "c1d0d273-9a23-425b-b389-0a38ca5fcc52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:23:37.222999 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.222972    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d0d273-9a23-425b-b389-0a38ca5fcc52-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c1d0d273-9a23-425b-b389-0a38ca5fcc52" (UID: "c1d0d273-9a23-425b-b389-0a38ca5fcc52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:23:37.223097 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.223013    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d0d273-9a23-425b-b389-0a38ca5fcc52-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1d0d273-9a23-425b-b389-0a38ca5fcc52" (UID: "c1d0d273-9a23-425b-b389-0a38ca5fcc52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:23:37.223097 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.223025    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d0d273-9a23-425b-b389-0a38ca5fcc52-kube-api-access-l944s" (OuterVolumeSpecName: "kube-api-access-l944s") pod "c1d0d273-9a23-425b-b389-0a38ca5fcc52" (UID: "c1d0d273-9a23-425b-b389-0a38ca5fcc52"). InnerVolumeSpecName "kube-api-access-l944s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:23:37.251229 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.251205    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:37.322156 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.322120    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1d0d273-9a23-425b-b389-0a38ca5fcc52-metrics-certs\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:23:37.322156 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.322154    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1d0d273-9a23-425b-b389-0a38ca5fcc52-maas-api-tls\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:23:37.322319 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.322163    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1d0d273-9a23-425b-b389-0a38ca5fcc52-openshift-service-ca\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:23:37.322319 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.322172    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l944s\" (UniqueName: \"kubernetes.io/projected/c1d0d273-9a23-425b-b389-0a38ca5fcc52-kube-api-access-l944s\") on node \"ip-10-0-136-121.ec2.internal\" DevicePath \"\""
Sep 12 01:23:37.358937 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.358907    2579 generic.go:356] "Generic (PLEG): container finished" podID="c1d0d273-9a23-425b-b389-0a38ca5fcc52" containerID="4d5e7cb2a248f01460a0b5e80b0a22386033e7a555e914a62faa575829f209bf" exitCode=0
Sep 12 01:23:37.359081 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.358949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2" event={"ID":"c1d0d273-9a23-425b-b389-0a38ca5fcc52","Type":"ContainerDied","Data":"4d5e7cb2a248f01460a0b5e80b0a22386033e7a555e914a62faa575829f209bf"}
Sep 12 01:23:37.359081 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.358952    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"
Sep 12 01:23:37.359081 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.358990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2" event={"ID":"c1d0d273-9a23-425b-b389-0a38ca5fcc52","Type":"ContainerDied","Data":"7d2623194d364c1fe5c417e68ccb9e18e13464408681b41b5cc8fee804e2e4b9"}
Sep 12 01:23:37.359081 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.359012    2579 scope.go:117] "RemoveContainer" containerID="4d5e7cb2a248f01460a0b5e80b0a22386033e7a555e914a62faa575829f209bf"
Sep 12 01:23:37.367894 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.367870    2579 scope.go:117] "RemoveContainer" containerID="4d5e7cb2a248f01460a0b5e80b0a22386033e7a555e914a62faa575829f209bf"
Sep 12 01:23:37.368202 ip-10-0-136-121 kubenswrapper[2579]: E0912 01:23:37.368161    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d5e7cb2a248f01460a0b5e80b0a22386033e7a555e914a62faa575829f209bf\": container with ID starting with 4d5e7cb2a248f01460a0b5e80b0a22386033e7a555e914a62faa575829f209bf not found: ID does not exist" containerID="4d5e7cb2a248f01460a0b5e80b0a22386033e7a555e914a62faa575829f209bf"
Sep 12 01:23:37.368270 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.368210    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d5e7cb2a248f01460a0b5e80b0a22386033e7a555e914a62faa575829f209bf"} err="failed to get container status \"4d5e7cb2a248f01460a0b5e80b0a22386033e7a555e914a62faa575829f209bf\": rpc error: code = NotFound desc = could not find container \"4d5e7cb2a248f01460a0b5e80b0a22386033e7a555e914a62faa575829f209bf\": container with ID starting with 4d5e7cb2a248f01460a0b5e80b0a22386033e7a555e914a62faa575829f209bf not found: ID does not exist"
Sep 12 01:23:37.381813 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.381763    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"]
Sep 12 01:23:37.385222 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.385202    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc46ff6dc-dkxj2"]
Sep 12 01:23:37.526355 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.526328    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"]
Sep 12 01:23:37.528245 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:23:37.528220    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b0c1ea_4465_4948_8e06_02de12faa80d.slice/crio-a40da36204d1fad59a9d4e2ca1a49ea9724596a0f44c9a94aca6d188b12f106c WatchSource:0}: Error finding container a40da36204d1fad59a9d4e2ca1a49ea9724596a0f44c9a94aca6d188b12f106c: Status 404 returned error can't find the container with id a40da36204d1fad59a9d4e2ca1a49ea9724596a0f44c9a94aca6d188b12f106c
Sep 12 01:23:37.590734 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:37.590702    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1d0d273-9a23-425b-b389-0a38ca5fcc52" path="/var/lib/kubelet/pods/c1d0d273-9a23-425b-b389-0a38ca5fcc52/volumes"
Sep 12 01:23:38.364567 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:38.364530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh" event={"ID":"a6b0c1ea-4465-4948-8e06-02de12faa80d","Type":"ContainerStarted","Data":"fd415ceacb3c9d42354c91a978c13495ca0446213bec747bb1bf16375ff28219"}
Sep 12 01:23:38.365035 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:38.364590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh" event={"ID":"a6b0c1ea-4465-4948-8e06-02de12faa80d","Type":"ContainerStarted","Data":"a40da36204d1fad59a9d4e2ca1a49ea9724596a0f44c9a94aca6d188b12f106c"}
Sep 12 01:23:38.365035 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:38.364637    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:38.383409 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:38.383355    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh" podStartSLOduration=1.9516859979999999 podStartE2EDuration="2.383336518s" totalImagesPullingTime="431.65052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:23:38.382183739 +0000 UTC m=+2443.301475526" watchObservedRunningTime="2026-09-12 01:23:38.383336518 +0000 UTC m=+2443.302628301"
Sep 12 01:23:44.375326 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:44.375292    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-749bf69647-vpbrh"
Sep 12 01:23:56.065504 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:56.065400    2579 scope.go:117] "RemoveContainer" containerID="7762dee6a6e8992037f25c3045821e63ad9b7300800c2e409e1fa2c9699ed416"
Sep 12 01:23:56.083032 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:56.073735    2579 scope.go:117] "RemoveContainer" containerID="4036b21c0437c77be1f4819160852a41eaef28c8be22daad8c8a5d032aacb5fa"
Sep 12 01:23:56.083032 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:23:56.080867    2579 scope.go:117] "RemoveContainer" containerID="94a7dfd34efc8a5ed740234da86046c48398056eb2ea8108ef4f25173c4d204b"
Sep 12 01:24:04.829066 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:04.829031    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d4q4m_d5ef8ce8-3d5e-4344-b605-b782c357bf1c/manager/0.log"
Sep 12 01:24:05.084443 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:05.084361    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f459795f7-ff5sd_d6ca53e2-99c0-4105-8d23-6a0e4b0329e7/manager/0.log"
Sep 12 01:24:05.701342 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:05.701309    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694bbb96f5-z5mv2_295af638-35e1-40a6-ab09-8f27213a7cbc/copy-manifests/0.log"
Sep 12 01:24:05.707961 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:05.707927    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694bbb96f5-z5mv2_295af638-35e1-40a6-ab09-8f27213a7cbc/manager/0.log"
Sep 12 01:24:06.190199 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:06.190171    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-749bf69647-vpbrh_a6b0c1ea-4465-4948-8e06-02de12faa80d/maas-api/0.log"
Sep 12 01:24:06.307314 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:06.307283    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819580-vmc6q_b1ff60c3-c329-4fa2-bbc8-f624a02dafe1/cleanup/0.log"
Sep 12 01:24:06.424368 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:06.424346    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819595-zsqp4_a6ae4a30-e750-45d0-8b72-93ef713187b7/cleanup/0.log"
Sep 12 01:24:07.472089 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:07.472056    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-5z5th_b8e6b77d-6517-43bb-8c21-61296874691d/manager/0.log"
Sep 12 01:24:07.839104 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:07.839035    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-cnzl4_a8a9200b-2ad5-41f7-8cb4-81e1cedfb143/manager/0.log"
Sep 12 01:24:08.596141 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:08.596112    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-47vv5_55eebaf5-c290-42c2-90ce-dd5934366f6c/discovery/0.log"
Sep 12 01:24:08.711786 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:08.711759    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f75b6b767-wrxh6_d2dd22ae-e545-4441-a1e0-1a1d1dc46079/kube-auth-proxy/0.log"
Sep 12 01:24:08.967354 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:08.967323    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-5hvwt_a081c880-b89f-4024-b8f6-e236eaf409f7/istio-proxy/1.log"
Sep 12 01:24:09.211864 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:09.211834    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-plxcj_f4fff3f1-7b21-4c2d-a4fc-33d2a733834e/payload-processing/0.log"
Sep 12 01:24:09.683267 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:09.683239    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-7kksj_770731cd-3368-43f7-8348-da4d3786353c/main/0.log"
Sep 12 01:24:10.150086 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:10.150008    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxpgc9_e81e26bb-c791-4118-8366-02743ce76d5e/main/0.log"
Sep 12 01:24:19.437411 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:19.437381    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nggqt_c34751eb-27f9-4061-8e37-d49bd4bb8bd5/global-pull-secret-syncer/0.log"
Sep 12 01:24:19.524422 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:19.524396    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ntsr9_fae396c5-d77f-4f6e-9881-b69beadbc497/konnectivity-agent/0.log"
Sep 12 01:24:19.588545 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:19.588515    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-121.ec2.internal_7cbe291fdf0e33f3b3d11babbfc8736a/haproxy/0.log"
Sep 12 01:24:24.732465 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:24.732432    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-5z5th_b8e6b77d-6517-43bb-8c21-61296874691d/manager/0.log"
Sep 12 01:24:24.840160 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:24.840132    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-cnzl4_a8a9200b-2ad5-41f7-8cb4-81e1cedfb143/manager/0.log"
Sep 12 01:24:26.650985 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:26.650958    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8xb2p_5071b60a-fd23-44b4-94a4-9fbebd984ebc/node-exporter/0.log"
Sep 12 01:24:26.679160 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:26.679134    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8xb2p_5071b60a-fd23-44b4-94a4-9fbebd984ebc/kube-rbac-proxy/0.log"
Sep 12 01:24:26.703675 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:26.703656    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8xb2p_5071b60a-fd23-44b4-94a4-9fbebd984ebc/init-textfile/0.log"
Sep 12 01:24:29.532812 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.532778    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"]
Sep 12 01:24:29.533172 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.533084    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1d0d273-9a23-425b-b389-0a38ca5fcc52" containerName="maas-api"
Sep 12 01:24:29.533172 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.533095    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1d0d273-9a23-425b-b389-0a38ca5fcc52" containerName="maas-api"
Sep 12 01:24:29.533172 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.533141    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1d0d273-9a23-425b-b389-0a38ca5fcc52" containerName="maas-api"
Sep 12 01:24:29.536654 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.536635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.539362 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.539340    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hgbmv\"/\"default-dockercfg-dwdhd\""
Sep 12 01:24:29.539473 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.539367    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgbmv\"/\"kube-root-ca.crt\""
Sep 12 01:24:29.539473 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.539368    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgbmv\"/\"openshift-service-ca.crt\""
Sep 12 01:24:29.547766 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.547738    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"]
Sep 12 01:24:29.629356 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.629322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv7dm\" (UniqueName: \"kubernetes.io/projected/bb0fe894-ed42-4007-8249-88d08119f559-kube-api-access-bv7dm\") pod \"perf-node-gather-daemonset-dtgp4\" (UID: \"bb0fe894-ed42-4007-8249-88d08119f559\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.629514 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.629375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb0fe894-ed42-4007-8249-88d08119f559-sys\") pod \"perf-node-gather-daemonset-dtgp4\" (UID: \"bb0fe894-ed42-4007-8249-88d08119f559\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.629514 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.629423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb0fe894-ed42-4007-8249-88d08119f559-proc\") pod \"perf-node-gather-daemonset-dtgp4\" (UID: \"bb0fe894-ed42-4007-8249-88d08119f559\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.629514 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.629456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb0fe894-ed42-4007-8249-88d08119f559-podres\") pod \"perf-node-gather-daemonset-dtgp4\" (UID: \"bb0fe894-ed42-4007-8249-88d08119f559\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.629514 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.629497    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb0fe894-ed42-4007-8249-88d08119f559-lib-modules\") pod \"perf-node-gather-daemonset-dtgp4\" (UID: \"bb0fe894-ed42-4007-8249-88d08119f559\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.730824 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.730796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb0fe894-ed42-4007-8249-88d08119f559-lib-modules\") pod \"perf-node-gather-daemonset-dtgp4\" (UID: \"bb0fe894-ed42-4007-8249-88d08119f559\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.730968 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.730847    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv7dm\" (UniqueName: \"kubernetes.io/projected/bb0fe894-ed42-4007-8249-88d08119f559-kube-api-access-bv7dm\") pod \"perf-node-gather-daemonset-dtgp4\" (UID: \"bb0fe894-ed42-4007-8249-88d08119f559\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.730968 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.730888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb0fe894-ed42-4007-8249-88d08119f559-sys\") pod \"perf-node-gather-daemonset-dtgp4\" (UID: \"bb0fe894-ed42-4007-8249-88d08119f559\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.730968 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.730902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb0fe894-ed42-4007-8249-88d08119f559-proc\") pod \"perf-node-gather-daemonset-dtgp4\" (UID: \"bb0fe894-ed42-4007-8249-88d08119f559\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.730968 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.730918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb0fe894-ed42-4007-8249-88d08119f559-podres\") pod \"perf-node-gather-daemonset-dtgp4\" (UID: \"bb0fe894-ed42-4007-8249-88d08119f559\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.731122 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.730973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb0fe894-ed42-4007-8249-88d08119f559-lib-modules\") pod \"perf-node-gather-daemonset-dtgp4\" (UID: \"bb0fe894-ed42-4007-8249-88d08119f559\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.731122 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.730984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb0fe894-ed42-4007-8249-88d08119f559-sys\") pod \"perf-node-gather-daemonset-dtgp4\" (UID: \"bb0fe894-ed42-4007-8249-88d08119f559\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.731122 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.730984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bb0fe894-ed42-4007-8249-88d08119f559-proc\") pod \"perf-node-gather-daemonset-dtgp4\" (UID: \"bb0fe894-ed42-4007-8249-88d08119f559\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.731122 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.731040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bb0fe894-ed42-4007-8249-88d08119f559-podres\") pod \"perf-node-gather-daemonset-dtgp4\" (UID: \"bb0fe894-ed42-4007-8249-88d08119f559\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.739096 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.739073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv7dm\" (UniqueName: \"kubernetes.io/projected/bb0fe894-ed42-4007-8249-88d08119f559-kube-api-access-bv7dm\") pod \"perf-node-gather-daemonset-dtgp4\" (UID: \"bb0fe894-ed42-4007-8249-88d08119f559\") " pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:29.848578 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:29.848507    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:30.116261 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:30.116234    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"]
Sep 12 01:24:30.118519 ip-10-0-136-121 kubenswrapper[2579]: W0912 01:24:30.118494    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbb0fe894_ed42_4007_8249_88d08119f559.slice/crio-a229488ec13d20e2d3a45ea64aa24cb7e726ed099a7ef68e74730c3bc27016e2 WatchSource:0}: Error finding container a229488ec13d20e2d3a45ea64aa24cb7e726ed099a7ef68e74730c3bc27016e2: Status 404 returned error can't find the container with id a229488ec13d20e2d3a45ea64aa24cb7e726ed099a7ef68e74730c3bc27016e2
Sep 12 01:24:30.593399 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:30.593365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4" event={"ID":"bb0fe894-ed42-4007-8249-88d08119f559","Type":"ContainerStarted","Data":"3c01be42dc4bcb45e3748a143bc5466f7c76855c7f66b9be7a5fc6f6816a6e8a"}
Sep 12 01:24:30.593800 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:30.593413    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4" event={"ID":"bb0fe894-ed42-4007-8249-88d08119f559","Type":"ContainerStarted","Data":"a229488ec13d20e2d3a45ea64aa24cb7e726ed099a7ef68e74730c3bc27016e2"}
Sep 12 01:24:30.593800 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:30.593442    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:30.609694 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:30.609647    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4" podStartSLOduration=1.609628691 podStartE2EDuration="1.609628691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:24:30.60880614 +0000 UTC m=+2495.528097927" watchObservedRunningTime="2026-09-12 01:24:30.609628691 +0000 UTC m=+2495.528920477"
Sep 12 01:24:32.239327 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:32.239294    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sbqdk_2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f/dns/0.log"
Sep 12 01:24:32.260196 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:32.260170    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sbqdk_2e96ed3a-ab87-4d37-a3a6-ccce53b09a3f/kube-rbac-proxy/0.log"
Sep 12 01:24:32.287073 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:32.287052    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6q7h9_64c9aebc-9e60-472a-891a-7853806abd9e/dns-node-resolver/0.log"
Sep 12 01:24:32.907614 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:32.907585    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7m4gn_305c7a25-c9a4-4f29-ae7a-ff9bf695f4b5/node-ca/0.log"
Sep 12 01:24:34.759812 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:34.759778    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-47vv5_55eebaf5-c290-42c2-90ce-dd5934366f6c/discovery/0.log"
Sep 12 01:24:34.784060 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:34.784014    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f75b6b767-wrxh6_d2dd22ae-e545-4441-a1e0-1a1d1dc46079/kube-auth-proxy/0.log"
Sep 12 01:24:34.867429 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:34.867400    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-5hvwt_a081c880-b89f-4024-b8f6-e236eaf409f7/istio-proxy/0.log"
Sep 12 01:24:34.871551 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:34.871532    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-5hvwt_a081c880-b89f-4024-b8f6-e236eaf409f7/istio-proxy/1.log"
Sep 12 01:24:34.951616 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:34.951579    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-plxcj_f4fff3f1-7b21-4c2d-a4fc-33d2a733834e/payload-processing/0.log"
Sep 12 01:24:35.950282 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:35.950250    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xbw45_b20511dd-b5f9-4302-b030-13bd00f31b12/serve-healthcheck-canary/0.log"
Sep 12 01:24:36.410403 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:36.410318    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4q7b5_e37e550b-9235-4dd2-83e4-5e6fee7d2a9a/kube-rbac-proxy/0.log"
Sep 12 01:24:36.431495 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:36.431468    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4q7b5_e37e550b-9235-4dd2-83e4-5e6fee7d2a9a/exporter/0.log"
Sep 12 01:24:36.459358 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:36.459331    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4q7b5_e37e550b-9235-4dd2-83e4-5e6fee7d2a9a/extractor/0.log"
Sep 12 01:24:36.606809 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:36.606777    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hgbmv/perf-node-gather-daemonset-dtgp4"
Sep 12 01:24:38.987880 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:38.987851    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-d4q4m_d5ef8ce8-3d5e-4344-b605-b782c357bf1c/manager/0.log"
Sep 12 01:24:39.419659 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:39.419588    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5f459795f7-ff5sd_d6ca53e2-99c0-4105-8d23-6a0e4b0329e7/manager/0.log"
Sep 12 01:24:39.733318 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:39.733287    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694bbb96f5-z5mv2_295af638-35e1-40a6-ab09-8f27213a7cbc/manager/0.log"
Sep 12 01:24:39.755880 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:39.755857    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694bbb96f5-z5mv2_295af638-35e1-40a6-ab09-8f27213a7cbc/copy-manifests/0.log"
Sep 12 01:24:41.111427 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:41.111333    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-47vv5_55eebaf5-c290-42c2-90ce-dd5934366f6c/discovery/0.log"
Sep 12 01:24:41.121038 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:41.121012    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f75b6b767-wrxh6_d2dd22ae-e545-4441-a1e0-1a1d1dc46079/kube-auth-proxy/0.log"
Sep 12 01:24:41.157559 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:41.157531    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-5hvwt_a081c880-b89f-4024-b8f6-e236eaf409f7/istio-proxy/0.log"
Sep 12 01:24:41.161858 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:41.161836    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-5hvwt_a081c880-b89f-4024-b8f6-e236eaf409f7/istio-proxy/1.log"
Sep 12 01:24:41.214252 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:41.214225    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-plxcj_f4fff3f1-7b21-4c2d-a4fc-33d2a733834e/payload-processing/0.log"
Sep 12 01:24:43.506356 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:43.506325    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-47vv5_55eebaf5-c290-42c2-90ce-dd5934366f6c/discovery/0.log"
Sep 12 01:24:43.526167 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:43.526140    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f75b6b767-wrxh6_d2dd22ae-e545-4441-a1e0-1a1d1dc46079/kube-auth-proxy/0.log"
Sep 12 01:24:43.565382 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:43.565337    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-5hvwt_a081c880-b89f-4024-b8f6-e236eaf409f7/istio-proxy/0.log"
Sep 12 01:24:43.570036 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:43.570012    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-5hvwt_a081c880-b89f-4024-b8f6-e236eaf409f7/istio-proxy/1.log"
Sep 12 01:24:43.631676 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:43.631649    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-plxcj_f4fff3f1-7b21-4c2d-a4fc-33d2a733834e/payload-processing/0.log"
Sep 12 01:24:47.381550 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:47.381519    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4nnlj_ea448e56-0b01-4890-a060-7a7de4a69b12/kube-multus/0.log"
Sep 12 01:24:47.411073 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:47.411047    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4l4j_8fd9311d-6922-42f5-8689-e768402de953/kube-multus-additional-cni-plugins/0.log"
Sep 12 01:24:47.431877 ip-10-0-136-121 kubenswrapper[2579]: I0912 01:24:47.431856    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4l4j_8fd9311d-6922-42f5-8689-e768402de953/egress-router-binary-copy/0.log"